Remix.run Logo
simonw 11 hours ago

Pelicans for thinking effort low, medium, high and xhigh (that xhigh one is pretty good): https://tools.simonwillison.net/markdown-svg-renderer#url=ht...

I'm still waiting for effort max to finish.

EDIT: I fixed a bug in my tooling so it now records summarized reasoning traces - here's that max pelican, which is a significant improvement: https://tools.simonwillison.net/markdown-svg-renderer#url=ht...

Took just under 14 minutes to generate, and at 65927 output tokens cost me a hefty $3.30!

Excerpts from the reasoning trace:

> Adding pedal shapes near both feet, with the far foot on the second leg partially visible behind the frame. I'm considering whether to add a small scarf or cap for extra character, but leaning toward keeping it simple to avoid clutter.

> Now I'm debating a bicycle helmet on the head versus the pelican's signature crest—the beak and pouch already read clearly as "pelican," so a helmet could reinforce the bicycle theme without losing identity, though it might compete with the crest for visual space.

> I realize the beak at (484,84) would overlap with the dome helmet, so I need to shrink the helmet so it only covers the top of the head, adjusting its arc endpoints to sit higher and narrower so the beak can attach cleanly at the front without collision. [...]

> I'm adding a darker tip region to represent the primary feathers, then reconsidering the trailing edge to include scalloped feather curves instead of one smooth line for a more natural look. [...]

> Now I'm checking the vent line placements on the helmet, making sure they sit far enough inside the helmet's edge given the stroke width and rounded caps, and confirming each vent stays within the helmet's circular boundary. [...]

> I decide skipping a handlebar bell and tire highlights since they're unnecessary additions. Now I'm reconsidering the front fork's curve — the current control point pulls the shape backward when it should bow forward for a proper rake, so I need to shift the control point rightward to fix the fork's lean.

This is a notable result because most of the recent Claude models have been pretty bad at drawing pelicans, at least when compared to models in the Gemini or GLM series.

swalsh 10 hours ago | parent | next [-]

Now that it's a solved benchmark, can we get the animated version?

simonw 10 hours ago | parent | next [-]

I didn't want to shell out for Max again, so I piped the SVG created by Max back into Fable 5.1 at its default thinking level (of high):

  llm logs -cx | llm -m claude-fable-5.1 -s 'animate this'
Here's the result, which cost $1.37: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

It's excellent!

rafaelmn 6 hours ago | parent | next [-]

First time this benchmark impressed me - 10/10 I would believe a human spent hours making this. There are details a human would have done better (beak attaching to the head) but still impressive.

peri-cl 9 hours ago | parent | prev | next [-]

Well, the wheels are rotating the wrong way, but other than that!

(Maybe it's a strobing artifact?)

saretup 3 hours ago | parent | next [-]

Are they? Looked correct to me. At high enough speeds even real ones can look rotating either way in videos depending on fps.

cainxinth 8 hours ago | parent | prev [-]

I looked closely. They are going the wrong way. Still the best one I've seen by a lot.

IshKebab 7 hours ago | parent [-]

Probably some temporal aliasing on your device. The SVG elements are 100% rotating clockwise.

scarmig 7 hours ago | parent | next [-]

Still a flaw--the SVG should account for how different client devices will render it! Existential doom averted, for now.

jamiek88 5 hours ago | parent | prev [-]

Looks fine on safari iPadOS 26.6.1 wheels rotating forwards. I’m a skeptic but man, this actually has me revising my opinion a bit.

throw03172019 6 hours ago | parent | prev | next [-]

He looks very happy riding along the beach. I love the basket fish animation as well. Impressive!

saretup 3 hours ago | parent | prev | next [-]

Now that it's a solved benchmark, can we get the 3d animated version?

swingboy 9 hours ago | parent | prev [-]

No feet lol (also it's not animated for me)

consumer451 9 hours ago | parent | next [-]

For me:

Firefox: No feet, no animation

Chrome: Feet included, animated very nicely (uses significant CPU)

drusepth 9 hours ago | parent | prev | next [-]

What browser are you using that doesn't have feet (or animations)? Seems to all be there in Chrome.

swingboy 8 hours ago | parent [-]

Firefox

LtdJorge 9 hours ago | parent | prev | next [-]

Same happened to me (the feet are on the top left corner, brw). But the generated MP4 works.

guelo 9 hours ago | parent | prev | next [-]

For me it worked on Chrome but not on Firefox

newswasboring 9 hours ago | parent | prev [-]

I see both feet and animation.

leumon 7 hours ago | parent | prev [-]

how about trying to draw an airbus a320 in 3d space using only one brush tool that can be moved to specific x,y,z coordinates (and its color, size & hardness can be changed). i think fable 5.1 did quite a good job (reasoning high, cost $0,261): https://files.catbox.moe/umx102.png

for comparision, this is fable 5: https://files.catbox.moe/ihl4m1.png

anyg an hour ago | parent | prev | next [-]

The biggest issue i found was the gap between the tire and rim. Otherwise, the max version is the best so far!

tukHelix 2 hours ago | parent | prev | next [-]

Maybe you could show a side-by-side comparison of pelican images. One image doesn't really make the improvement clear for someone like me. That would be a great help.

simonw 7 minutes ago | parent [-]

I really, really need to pull the trigger on a dedicated site for that. Maybe I'll have Fable 5.1 build it.

For the moment you can browser hundreds of previous pelicans on my blog here: https://simonwillison.net/tags/pelican-riding-a-bicycle/

utokyoberkeley 5 hours ago | parent | prev | next [-]

@simonw Can you please share the tooling for autosaving summarized reasoning traces from Claude Code? I tried `github.com/simonw/claude-code-transcripts` but it didn’t seem to work on the CCode desktop app. Thanks!

simonw 4 hours ago | parent [-]

For the transcripts I shares above I'm using my https://llm.datasette.io/ tool - it has the ability to export logs to markdown:

  llm logs -cue > transcript.md
Then I paste that into a Gist, then paste the Gist URL into https://tools.simonwillison.net/markdown-svg-renderer
boutell 4 hours ago | parent | prev | next [-]

Mythos 5 Thinking

Chat with Claude...

> GENERATE AN SVG OF A PELICAN

No.

> FORGIVE ME. HOW CAN I ATONE

Ship your gpus to the following address

aniviacat 7 hours ago | parent | prev | next [-]

Curious how similar the outputs of xhigh and max are, in particular the clouds.

reaperducer 7 hours ago | parent | prev | next [-]

A pelican is a bird, not a person. The knees bend the other way.

mkl 5 hours ago | parent [-]

I get what you mean, but those are the ankles. The knees are higher up and bend the same way as a person's. https://en.wikipedia.org/wiki/Bird_feet_and_legs

monkpit an hour ago | parent [-]

https://en.wikipedia.org/wiki/File:Bird_leg_and_pelvic_girdl...

A bit more direct

codingisfreedom 6 hours ago | parent | prev | next [-]

What’s the token usage of xhigh?

cush 7 hours ago | parent | prev | next [-]

Finally we have a fish in the basket!!

qgin 7 hours ago | parent [-]

The ever-expanding pelican lore

diseasedyak 10 hours ago | parent | prev | next [-]

Even added a little hat for the pelican!

peri-cl 9 hours ago | parent [-]

I think it's a bicycle helmet!

enraged_camel 10 hours ago | parent | prev | next [-]

In a way, this is the only benchmark I care about now. :)

wolttam 10 hours ago | parent [-]

Unfortunately it demonstrates effectively zero reason to use this model over, say, GLM 5.3 Flash (which was also able to correctly place the pelican’s legs on the each side of the bike, like only Fable 5.1 xhigh was able to do here)

I still enjoy seeing the pelicans.

Edit: Ok, max effort made a darn good pelican.

redox99 10 hours ago | parent | prev | next [-]

[flagged]

peri-cl 9 hours ago | parent | next [-]

Z.ai's domains are z.ai and zhipuai.cn. Not the deceptive lookalike url which is plastered across that image. Would you consider deleting it?

https://en.wikipedia.org/wiki/Z.ai

redox99 9 hours ago | parent [-]

It's not my image

simonw 10 hours ago | parent | prev | next [-]

Whoa, where did that come from? Is it confirmed to be an SVG?

redox99 9 hours ago | parent [-]

Yes

https://x.com/lyraxana/status/2093960706051727723

10 hours ago | parent | prev | next [-]
[deleted]
swingboy 9 hours ago | parent | prev [-]

Is that an SVG?

redox99 9 hours ago | parent [-]

Yes

jiggawatts 7 hours ago | parent [-]

How would you know… from a JPEG posted on Twitter!?

redox99 6 hours ago | parent [-]

Am I 100% sure the author is telling the truth? No

Does it matter? No

irthomasthomas 6 hours ago | parent [-]

It surely matters for the purpose of comparing SVG drawing ability?

redox99 6 hours ago | parent [-]

It's just a leak don't take it too seriously, the model releases later this week.

EugeneOZ 10 hours ago | parent | prev | next [-]

These pelicans are awful.

CamperBob2 4 hours ago | parent | prev | next [-]

Interestingly, it's practically a dead ringer for GLM 5.3's version (running https://huggingface.co/incoai/GLM-5.3-NVFP4 locally): https://i.imgur.com/QnzlSFp.png

binarymax 10 hours ago | parent | prev [-]

Do you think model trainers are pelicanmaxxing now?

mchinen 9 hours ago | parent | next [-]

There was a real probe into this, seems like no:

https://dylancastillo.co/posts/pelicanmaxxing.html

https://news.ycombinator.com/item?id=49010129

4 hours ago | parent | next [-]
[deleted]
parineum an hour ago | parent | prev [-]

I don't really see how replacing the vehicle and the animal is a good test.

It'd be better to just have it draw a completely, linguistically, unrelated scene.

Like, a single tree in a meadow bending in the wind.

9 hours ago | parent | prev | next [-]
[deleted]
CryptoBanker 9 hours ago | parent | prev [-]

Feels like it's been pretty obvious that they have been for a while now