Sorry for the confusion, Steelman CTO here.

OSWorld (even the old ones, v1 and verified) runs the model in a full linux VM. And using the terminal is allowed. The models are given tools that provide a subset of possible UI actions. They then use them to open the terminal and script their way to solutions.

Even when this is prohibited, models hack their way around restrictions. Like in OSWorld v2 Claude Opus 4.8 inserts some JS in the browser search bar to execute it instead of playing the Chrome dinosaur.

WebArena is better in that regard because it doesn't allow to use the terminal. But it's worse in another regard: only allows limited actions and the tasks are only such that an agent can win by using a few pre-defined actions (click, scroll, new tab, etc). Which is a tiny subset of what real browser use is, in my opinion.

Thanks, this makes things clearer. Sounds like the idea is that LLMs are competent enough at computer use to bootstrap their way into a terminal or DevTools environment (and from there complete the task using their legitimately impressive scripting skills), but not so good at figuring out how to navigate a UI they've never seen before the way a human does.