Hang on, can someone help me understand how these benchmarks work? The post alleges that at least some computer use benchmarks are broken because the model is making direct CDP calls instead of actually interacting with the UI. But the major frontier models' computer use interfaces don't work that way; the model only gets access to a virtual display, and can only provide virtual keyboard and mouse inputs. There are separate browser-use interfaces that offer a CDP connection, but presumably you wouldn't use those if you're specifically designing a computer use benchmark? Is the post just claiming that a lot of impressive demos that people think are of computer use are actually of browser use? I wish it'd been clearer about this.

Sorry for the confusion, Steelman CTO here.

OSWorld (even the old ones, v1 and verified) runs the model in a full linux VM. And using the terminal is allowed. The models are given tools that provide a subset of possible UI actions. They then use them to open the terminal and script their way to solutions.

Even when this is prohibited, models hack their way around restrictions. Like in OSWorld v2 Claude Opus 4.8 inserts some JS in the browser search bar to execute it instead of playing the Chrome dinosaur.

WebArena is better in that regard because it doesn't allow to use the terminal. But it's worse in another regard: only allows limited actions and the tasks are only such that an agent can win by using a few pre-defined actions (click, scroll, new tab, etc). Which is a tiny subset of what real browser use is, in my opinion.