Here are pelicans for thinking levels low, medium, high, and xhigh: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

All four levels have a correctly shaped bicycle frame. The differences between the pelicans aren't huge, but the xhigh one has a better beak.

I haven't managed to get one for level "max" yet, it hit the limit of 128,000 cap for output tokens while it was still reasoning about the question!

Max started its thinking trace like this:

> This is a classic test request, so I want to plan out a well-composed pelican with its distinctive beak and pouch riding a bicycle with proper wheels, frame, and pedals, set against a simple sky and ground backdrop.

So that failed attempt on max cost me $2.56.

I ran this using my llm-anthropic plugin:

  uv tool install llm
  llm install llm-anthropic --upgrade
  llm keys set anthropic
  # paste key here

  llm -m claude-opus-5.5 -o thinking_effort low "Generate an SVG of a pelican riding a bicycle"

  # Then to save the markdown logs
  llm logs -cu > logs-with-usage.md

> This is a classic test request

Isn't this basically the model admitting it was trained on this? Otherwise why would it think a pelican svg is a usual request?

>I haven't managed to get one for level "max" yet, it hit the limit of 128,000 cap for output tokens while it was still reasoning about the question!

Off to a _great_ start...

Also interesting this somewhat mirrors my recent experience with Opus 5--too much effort and it starts looking for things to do and invents requirements that never existed

> The differences between the pelicans aren't huge, but the xhigh one has a better beak.

If you look carefully, everything except the last pelican has the two legs both in front of the crossbar as if the legs are all on one side of the bike.

The last pelican gets this correct.

It is interesting that Fable 5.1 max [1] which also produced a decent pelican with 65k output tokens compared to 5.5 running out of 128k output tokens tells us something about the new models token usage propensity despite this being a sample of 1.

Fable 5.1 27 input, 65,927 output

Opus 5.5 27 input, 128,000 output (128k thinking tokens) - incomplete

[1] https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

surprising it took until xhigh before it did the legs properly - all the ones before that have both legs on the near side of the bike.

I agree, not a huge difference here. They eyes and ... hat? on high are out of place so I'd argue that's the worst one, but it takes xhigh before we get legs and bike ordering correct.

Xhigh is very, very solid.

I do always wonder why every model does the exact same 'from the side, going right' perspective though. Seems oddly convergent.

With the frequency of model releases, pelicans seem to have become a part-time job for you. But unpaid :/

I guess that means you are officially the creator of a "classic" LLM test. Congrats!

Heh. Pelican-benchmaxxing is real.

> I haven't managed to get one for level "max" yet, it hit the limit of 128,000 cap for output tokens while it was still reasoning about the question!

Does anyone else find this outright insane? It wrote the equivalent of a full-length novel, just to sit in the question of planning a few dozen shapes.

Academia is going to love this :)

I don't think this is very helpful to assess the LLMs capability levels anymore

Adding the tuft and improving the beak.. PelicanBench is becoming a solved problem.

Not as good as Astra or Fable 5.1 on this test as far as I can see. I wonder if any benchmark exists for artistic taste, visual sophistication etc. I think your Pelican test does touch on these aspects of a model and is useful for developers trying to build rich digital experiences (includes games, interactive websites and apps). These benchmarks are subjective so it may not be easily established and will have polarized reactions before it gains legitimacy. May even need human judgement layers adding to the cost of running it.

Lmao each one gets worse as the effort increases.

[dead]

[dead]

This benchmark is useless and should die. LLMs have likely trained on it, it's too easy to game by training specifically for it, & it doesn't mean much