I generated pelicans riding bicycles on both thinking level low and thinking level high:

https://gist.github.com/simonw/68560eddb0b268a8417f80ceb7304...

The high one is notably better - the bicycle frame is the correct shape, unlike thinking level low.

For comparison, here's Opus 4.7: https://gist.github.com/simonw/afcb19addf3f38eb1996e1ebe749c...

> the bicycle frame is the correct shape

No, the handlebar is wrong. The handle bar is rotating the frame instead of rotating the front wheel. The handle bar should be mounted on the same line as the front wheel is.

Hopefully 4.9 will read my comments :)

I bet someone shares this link every time you post about bicycles, but since I didn't see anyone share it yet in this thread, I'll take the opportunity to do so:

https://www.gianlucagimini.it/portfolio-item/velocipedia/

Turns out even humans can be pretty bad at drawing bicycles :)

Here's pelicans in all of the thinking levels - low, medium, high, xhigh, max

https://tools.simonwillison.net/markdown-svg-renderer#url=ht...

Glad to see that the "high thinking" level adds a helmet. Always a smart choice.

You've peed in the pool Simon, this has to be a part of the internal evals by now! You got to try something new - maybe a panda in a canoe?

I actually like the 4.7 the most, interestingly enough. Not like you can "objectively" weight artistic output like this.

I really like that thinking level high gave the pelican a helmet.

Hey simonw I love your test, do you think using thinking level "max" makes sense for this test? I would love to see the results about it.

I find the most miraculous thing about 4.7 to be that the pelican is facing left, wonder why the right facing everything is so ubiquitous in these images.

Simon, is your pelican test really captures differences among models or should you at least try like 10 times or something to average the random effects

The vast majority (if not all) of these make it impossible to turn, among other fun things. Only out of curiosity, have you tried prompting further with how a bike must operate to see if it does the right thing?

thanks for always providing this very much on time. I'm wondering what the next, harder challenge could be? Maybe some animated svg?

For comparison, what's GPT-5.5 producing today?

That little red hat on hard mode is sending me. 4.8 has whimsy

Is the "opossum riding an e-scooter" benchmark in the works for Opus 4.8? ;)

Eventually the frontier model folks are going to pick up on your pelican on a bike test and bake-in flawless results for that particular request.

Am I allowed to say that pelican's little helmet is adorable? I can't provide a strong computational proof, or even a shred of anecdata...

...but that pelican's little helmet is adorable.

4.7 reigns supreme IMO.