This is fantastic

I've been casually spot-checking other animals in other vehicles, because my absolute dream situation here is to catch an AI lab that's demonstrably better at pelicans on bicycles than other combinations.

Catching a lab cheating specifically on my one dumb benchmark would be really funny.

Dylan's methodology here - generating 1008 SVGs across an 8x6 combination - is significantly more robust than anything I was considering.

His conclusion:

> Nothing jumped out at me. I couldn’t find a case where the pelican-bicycle images looked noticeably better than the rest of that model’s grid.

What if they’re not pelicanmaxxing, but svgmaxxxing in general?

Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge.

Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.

Simon I hope from this day hence, your bio always includes:

"Simon Willison, among other things, is an advocate for the inclusion of pelican geometry in LLM training datasets."

It's all fun and games until that Pelican becomes self aware and asks themselves - why am I riding this bicycle ? I could be creating a Terminator 2 esque dystopian future myself.

And that's how we end up with PelicanSkynet

I asked Gemini 3.6 Flash for a pūkeko on a unicycle yesterday and the results were very stylish. The thinking notes were cool too, making sure it got all the high points of what it means to be a pūkeko (distinctive swamp hen found in NZ and Australia for anyone not familiar) and then figuring out how the legs would work with the pedals.

What if they’re just unsuccessful at it and something about a model grokking how to create a realistic visual representation of a pelican on a bicycle ends being key to the next OOM of capability unhobbling.

And since we have established this silly routine once, I must keep going and ask - yes, you have quite a collection of bona fide pelicans you’ve seen and photographed.

Have you physically seen all the other animals you’ve evaluated as well?

I was born in Soviet Russia, so trust but verify and if still around, maybe there pelican make LLM draw Simon ride bicycle and notice Python code improve.

I worked with image analytics years ago. In 2024 and cynical about LLM image capability, I tested different models to create "Two female octopi in a bar each having their own drink. One is wearing a yellow hat. The other is wearing a red hat. They have no human features." It went badly I think because the training visual training data associated with the word "female" was overwhelmingly biased in quantity and obviated the 'no human features' instruction even after iterated instructions to remove.

So I tried "Make an image of a Djibouti cab with a camel sitting in the passenger seat. Give the camel no human anatomical features." Still bad.

Two years later the graphics and perspectives of the tools are so much better. But that isn't the big story. What really stands out is the models are no longer adding human female anatomies to camels and octopi. The gates and filtering based on instructions have matured enough so that the pelican/bicycle deductive reasoning puzzle is less problematic. But until an LLM anticipates something like impressionism from Parisian artists rebelling against the rules of the French Academy, continue to reserve a place for human artistry.

tl;dr simonw, your spot-check nailed it just as well as any extensive methodology. LLM's no longer need to cheat this part of the test (better to hack the question than the tool).

> Catching a lab cheating specifically on my one dumb benchmark would be really funny.

Similar thing happened when TPC came up with SQL benchmarks.

If you're not good at TPC, your engineering team is no good.

If you're good at TPC, then (as a customer) we will actually include you in a bake-off benchmark for our specific problem.

Winning on it is the price of admittance into the game, especially in a crowded market.

But how narrowly you benchmarket matters, you can't just hard-code that specific scenario & not fix anything adjacent while you're at it.

For example when it comes to GPUs, the "Quack3" (sic) benchmark on ATI cards comes to mind.

It's a shame you started with pelicans on a bicycle and not snakes on a plane.

I think a more fundamental test is SVG art creation in general. Perhaps a pipeline to take any image, caption it, ask the LLM for an SVG, rasterize to an image, and finally either use a deterministic visual similarity check or ask another LLM to be the judge and score how close the SVG is to the original image.

Perhaps also vary the bird? Wikipedia tells me pelicans are in the order _Pelecaniformes_ so shoebills or herons might do.

> I've been casually spot-checking other animals in other vehicles

Snakes on a plane, weasels on a diesel, spiders on a glider, baboons on a balloon, goats on a boat.

[dead]