Every step here has an oracle: wall-clock, the profile, pass or fail from the verifier. I had an agent-built app audited task by task, 10 came back done and 7 worked, and the three misses were the ones needing a credential or a setting on someone else's dashboard. Nothing in the loop could tell the agent it had failed, so it said done and moved on.

▲

phoghed 4 hours ago | parent [-]

I’ve tried it on simple UI tasks. Give it screenshot to work towards (or figma MCP), let it get screenshots from chrome to check its work.

The existing models are surprisingly bad at it.

▲

iinnPP 4 hours ago | parent [-]

It's really difficult to understand what your definition of surprisingly bad is. What was it continuously having problems with?

	▲	phoghed 3 hours ago \| parent [-]
		Just reporting work as done that’s not even close mainly. Then if you manually screenshot both and ask if they are the same it’s like “yeah looks great boss”. The visual understanding of the models is just leagues worse than their text/language understanding. Or guessing colors rather than sampling from the image or pulling from figma is another stupid thing they do constantly.