I'd say it's working great for its intended purpose. Keeps Simon on top of all these threads and funnels traffic to his site.

Was it ever a good test? How do you even objectively assess what a good pelican on a bike is anyway?

Variations of this comment have been posted for over a year. The pelican has now morphed into part of HN culture rather than a legitimate benchmark, but it's still valuable as a meme.

Simon has addressed this on virtually every new model release. He also has unpublished alternate prompts. But the larger point is: this is a fun experiment, not a serious and objective benchmark.

I just run my own benchmark for "draw an SVG with $animal driving $vehicle". I won't post my choice of animal and mode of transport, but there are plenty of uncommon combinations to choose from. So far it's a fun and visually intuitive benchmark that does seem to correlate with model capabilities

I don't know. Just looking at the bike frames (specifically the fact that the AI generated bikes have rather unsteerable front forks), it's clear to me that frontier labs aren't spending much time tuning models to make bikes look coherent, which I assume is an easier task than making a pelican riding a bike look coherent.

It was a completely useless test even before the labs trained for it.

I've seen this reply to Simon's benchmark for 2 years running now, and yet you still see improvements and objectively-bad results over time from new releases, even when I'm sure every frontier AI team has/had a person at least partially dedicated to better bicycle-pelican SVG outputs. Alas.

I also look for this reply because i like seeing the follow-up reply saying that this is not a benchmark anymore because labs have gotten it in their training data.

that reply never failed to come it's basically a meme at this point

This is all we need, that moment the Pelican put the leg behind the frame, we are all doomed.

Now someone post the link about how it’s impossible for humans to draw a bike from memory.

Not as much as Qwen, since apparently 3.6 35B surpassed Opus 4.7 https://x.com/simonw/status/2044830134885306701

Probably none. They probably have much better targets to optimize for than an SVG pelican or even SVGs in general.

Or the hands not being wings

Pelicans hate biking in the rain (as do I).

You can't tell someone to "get a life" while taking the effort to create a burner account for the sole purpose of insulting someone.

I don't really consider that a great benchmark anyway and we really need better ones that are objective instead of these mostly performative and cheatable and also available in the training set.

Simon's pelicans are an institution. Are you trying to get banned. Lmao.