Reimagining the mouse pointer for the AI era

AbuAssar

so Google will be monitoring whatever on the screen continuously or only when the user say the magic words (this, that, here, there)?

loaderchips

It's beautiful how the human mind can take something very obvious but overlooked and make it into this fantastic innovation. Fab stuff.

SirFatty

It only took Google and their AI offering to come up with Graffiti.

themafia

> We’ve been exploring new AI-powered capabilities to help the pointer not only understand what it’s pointing at, but also why it matters to the user. We couldn't quite track you well enough before. So we're fixing that under the guise of "AI powered capabilities."

iridione

Interesting! I wonder how UI will evolve in the long-term? If there are browser-use/computer-use and clicky-clones automating pointer actions, do we really need complex UI anymore? If yes, when?

mvdtnz

Both of the text based demos would have been simpler and faster with traditional mouse and keyboard interactions. What is the AI adding?

mcookly

I wonder what sort of monstrous power would be unleashed if Google used Plan9 as a foundation.

tintor

Of course, it isn't a Google Demo, if you can't use it to book a table at restaurant. (shown at the bottom of the page)

LocalH

do not want

jaccola

This seems like one of those things that is usable infrequently enough to be forgotten/poorly developed/never used. (Even before accounting for the actual failure rate of the LLM which will be none-zero). Perhaps a text box and file upload isn’t the perfect interface for every use case but it is versatile which is a huge barrier to overcome.

jinkuan

being able to make precise edits would be huge for AI

OtomotO

Like a dream come true... Nightmares are dreams as well and this is a nightmare like Windows Recall. Technically wonderful though.

nolist_policy

Wiggle at CAPTCHAs, wiggle at Termux, wiggle at Emacs, wiggle at the Godot Editor, wiggle at my remote desktop. (Not going to happen)

jpatten

Reminds me of Put That There https://m.youtube.com/watch?v=RyBEUyEtxQo

kjellsbells

I sense a privacy problem brewing. It reminds me of Microsoft Recall in the sense that some portion of the screen is going to be continuously transmitted outside of the users control. What happens when someone browses something very private (planning a surprise engagement. looking at medical data. planning a protest)? All that data gets slurped to google and subject to a warrant or discovery or building your advertising fingerprint. Maybe the idea is that the data is sent to AI only when you right click, but that seems like a very thin firewall that a product manager will breach in the interests of delivering "predictive AI" via some kind of precomputed results.

simondw

Maybe I'm misunderstanding, but what is new about the pointer itself? Seems to be functionally the same as selecting + tooltips / context menus.

SirMaster

Thanks, I hate it

why_at

My first impression coming away from this is skepticism. Anything with voice controls for routine use is a pretty tough sell. Doing this when you're not completely alone would be annoying to everyone around you. Most of their examples seem like they could have been done with a right click drop down menu so they don't really need to "re-invent the mouse pointer". So is this thing talking to Google's servers all the time for the AI integration? So it won't work if you're not connected to the internet? Privacy concerns are obvious; now Google wants to have an AI watching literally everything you do on your computer? Does it cost the user anything for the LLM use? If it's free will it stay free forever? That's quite a lot to give away if they're expecting people to use it to change a single word like in one of their examples. I guess they're expecting to make the money back by gathering data about literally everything you do on your computer. There might be a killer app for AI integration with personal computers that has yet to be invented, but this doesn't look like it.

brgsk

what the hell is going on at google