I've worked on a LLM-based visual validation framework for mobile layouts of different websites. The results are pretty flaky, even with Claude models. I've actually seen better performance with older ones like Meta Llama 4 or Amazon's Nova (the solution was running on AWS Bedrock).

I've since refactored it to first do some deterministic checks using CSS, to determine visual glitches like overlaps. Only after do I use LLMs to figure out if text is broken, loading spinners still display, or other such things where models are better.

For deterministic tasks like pixel-by-pixel comparison, I don't think LLMs are good enough (for now, at least).

I've done VHS (terminal recording) rewrite in rust built on libghostty called Betamax [1], and then threw it at creating a TUI for jujutsu. This worked pretty well, but is admittedly a much simpler task than a web ui. The harness was able to build out a feature, confirm it worked, find bugs in what it expected vs saw and fix them with evidence. Web is many more pixels and more nuanced things that have to do with how various elements relate.

I speculate that maybe the OpenAI models have the edge here over Claude's (while still being terrible at web UIs), as the training necessary to handle computer use stuff is heavily image processing based. I wouldn't bet on either being bad for much longer though.

[1]: https://www.joshka.net/betamax/

It might help to consider how stable an arbitrary human would be at this task.

User interfaces are like pieces of artwork. You really don't know until it's finished, and even then it's not for everyone.

LLMs are most effective when working with human parsed representations of websites. They are terrible at doing the actual parsing.

They're also good at working on a relative basis. One png screenshot of the viewport is a difficult hill to climb. The difference between two screenshots is much more informative. Deltas usually tell a much bigger story than dead reckoning against the base world model.