You can't solve computer use by ignoring the interface

dzonga

that's what a lot of 'a.i' companies miss - unless you've formerly QA devs deployed as Forward Automation Engineers - writing scripts to automate most tasks done by humans in a service economy - by the time the a.i bubble busts it will all about the promise about a.i to transform the economy while it was near achieved. browser automation, RPA etc can bring us closer to agentic workflows since most software doesn't provide an API.

speak_plainly

The little pencil effect is a nice touch.

juancn

You lost me at the annoying trail on mouse movement. You advocate for good UI and immediately make some crap that makes it hard to read/use your site. It's kind of ironic. Please stop messing with: selection, mouse cursor, copy/paste, etc.

nvch

Looking how much agents like to use and push to have functional a11y trees, we may accidentally solve accessibility as well

gandreani

Meta: I like that the first comments are not about the model at all but the website's mouse trail. One is in favor and one is against. Sums up society pretty well hahaha. On the model. Really cool that it can act every 70ms. That's more than 10x faster than TTFT in most inference APIs. It HAS to be a local model. No idea which family or which one it is or what hardware it runs in. These would be nice details to tease!

kvark

We have 2 paths here: - either evolve our UI systems for computers and let them operate efficiently, i.e. converting everything to API calls . Frontier labs are pushing for this, starting with the Chrome plugin that Claude Cowork mandates. - or make AI systems use the screen like humans do - what the article is arguing for Both of these paths are possible. One of them is much more efficient than the other in the long term. Another one may be inevitable.

luciana1u

every decade someone rediscovers that the bottleneck was never the model, it was the mouse. we built APIs to avoid this exact problem and then asked the AI to ignore them and click buttons like an intern

charcircuit

Is no benchmark properly sandboxed? It feels like every single the logs are provided for a benchmark run the LLM is cheating in a way that should have been clearly blocked by a sandbox or the harness.

ameliaquining

Hang on, can someone help me understand how these benchmarks work? The post alleges that at least some computer use benchmarks are broken because the model is making direct CDP calls instead of actually interacting with the UI. But the major frontier models' computer use interfaces don't work that way; the model only gets access to a virtual display, and can only provide virtual keyboard and mouse inputs. There are separate browser-use interfaces that offer a CDP connection, but presumably you wouldn't use those if you're specifically designing a computer use benchmark? Is the post just claiming that a lot of impressive demos that people think are of computer use are actually of browser use? I wish it'd been clearer about this.

drob518

The article makes the point that on some benchmarks the AIs worked around the UI and then says that sometimes APIs won’t be available. The unwritten implication is that then we’ll be in trouble. But will we? If an API doesn’t exist, is the AI still able to perform the task? That’s not covered in any detail, nor is there any direct comparison between the frontier AI’s performance on these tasks and the model being sold by the company. At some level, I don’t fault the AI for taking the API path if it’s available. In fact, I’m impressed that it found the API and used it correctly. That doesn’t seem like an argument that the sky is falling.

haburka

Can AI play OSU? Well finally we have an answer and it’s just barely. Honestly the demo at the bottom is really surprisingly good since it’s realtime

_doctor_love

I wonder if we will see people building these kinds of computer use systems to augment traditional QA. What I do not have my mind wrapped around at the moment is how agents might reason across the different states of the interface. It seems like it should be possible to build some kind of state machine for this?

phn

Computer use itself is evolving as CLIs and APIs are important again to enable AI use. We'll certainly converge at a happy medium, CLIs, APIs and text for AI, and maybe some form of dynamic interface that the AI builds/adapts on the spot for the exact task you want to execute, on the environment you're at. And of course, good ol' purpose built tools/UIs as we do today, when it makes sense.

heaney-555

Why is this article based on a chart that has GPT-5.5 and Claude Opus 4.8 as the newest evaluated models? Those are very old now. There have been massive improvements in computer use in GPT 5.6 and Claude 5.

deterministic

The mouse trail is really annoying! Not a good example of how to do UI.