Well, there’s a bigger issue of AIs not doing what we tell them to do, but I think that’s orthogonal to this UI issue. If you want to fault the model for not obeying the prompt or for a poor solution, then fine. My point is, it solved the problem. If the API isn’t there, then that eliminates the API as a solution, but it says nothing about how well the model can solve the UI problem. Perhaps the model can’t solve it. If we really want to benchmark the ability of models to use human UIs to solve problems, then perhaps we need to choose benchmarks that don’t have APIs available such that the model cannot get “creative” in any way and must use the UI as part of the task. More simply, maybe the model isn’t the problem; maybe the benchmark designer is.

▲

mpavlov an hour ago | parent [-]

> If we really want to benchmark the ability of models to use human UIs to solve problems, then perhaps we need to choose benchmarks that don’t have APIs available such that the model cannot get “creative” in any way and must use the UI as part of the task. More simply, maybe the model isn’t the problem; maybe the benchmark designer is.

That's a valid point, yet it's hard to blame authors of OSWorld and ALE. They created an env for benchmarking long horizon task completion to be as close to real computer as possible. And for this goal CLI/API access is generally useful, yet when the model not defaults to it for the majority of subtasks.

There're benchmarks that would measure UI literacy (Webgames Benchmark is one). But they are far from the task we want to benchmark in the end.

	▲	drob518 42 minutes ago \| parent [-]
		Sure, “blame” is a strong word. My point isn’t that anyone deserves actual blame, but just that the models are proving creative. We already know that they will comment out unit tests to make them “pass,” for instance. If we want to focus on a particular skill or behavior, we need the tests to highly constrain the model. If we don’t create our benchmarks like that, shame on us.