Image to html comparisons of Fable 5.2 and Opus 5.5 (the previous SOTA for this test):

Fable 5.2: https://html.non.io/lcars-fable-5.2/

Opus 5.5: https://html.non.io/lcars-opus-5.5/

Source designs: https://diffui.ai/app/canvas/5093e689-1e74-4f26-b632-2a4500f...

Thoughts:

The best implementation wrt being as true as possible to the source design. It nails the flowing svg shapes. Icons are more accurate to the design - note the difference between the starship svg in the "Legendary platforms" card between Opus 5.5 and Fable 5.2. The 3js planets outside the windows are also really well done, especially the bottom one which has a nice background glow to it.

It isn't without issues though - the flicker effect it added to the S shaped svg that weaves through the page is distracting in my opinion, and I actually thing the motion animation in the Opus 5.5 one is better overall. Still, it nails the implementation of this page, but it's no surprise considering how good Opus 5.5 was at this.

Disclaimer / reminder: the objective of this test isn't to have a well designed webpage, it's a synthetic test that pushes the limits of LLM's vision capabilities. Traditionally LLMs have issues with smooth curves, svg animations, and non-boxy layouts, which this design pushes the limits of.

▲

docheinestages an hour ago | parent | next [-]

I wonder if any of the LLMs are smart enough to verify their work the way humans model 3D objects, i.e. by superimposing the original design on top of their work and checking for differences.

	▲	jjcm an hour ago \| parent [-]
		This is actually something I'm working on as part of a more formalized version of this benchmark, both as a tool to assist LLMs in doing this (especially for ones with a more poor vision model), but also to properly measure how well they've done. Still in progress, but early demo showing difference zones: https://non.io/video/fidelity-lens.mp4

▲

user43928 2 hours ago | parent | prev | next [-]

There's something wrong with the Fable 5.2 version, the Customizing button doesn't work for me in Chrome on Desktop.

	▲	jjcm 2 hours ago \| parent [-]
		That's me versus fable. I only told fable to build out the first page rather than the full app that's in the designs (was approaching the limit of my claude plan).

▲

enlyth an hour ago | parent | prev [-]

There's some very obvious mistakes in the Fable 5.2 version like the "Start Customizing" button overlapping under the colorful border lines, and the "EXPLORE WITHOUT LIMITS" title being cut off at the bottom.

This would've been caught by a junior web developer or just an everyday Joe glancing at the page for a few seconds

I guess AGI remains two weeks away..

▲

jjcm an hour ago | parent [-]

I've been surprised at how long it's taken for this particular test to become fully saturated. So far there hasn't been a model that's able to perfectly one shot it.

	▲	vatsachak 11 minutes ago \| parent [-]
		This just proves that LLMs are way more jagged than humans and are a tool...for now