Here's 'Generate an SVG of a ring-tailed lemur riding an electric scooter' at reasoning level max: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

Quote from the thinking trace:

> I’m thinking about how a helmet would obscure lemur ears, but using an electric scooter helmet seems responsible.

It's pretty solid - face is a little wonky but excellent tail and scooter.

I would expect it happens more organically where people’s discussion of the benchmark and posted results find their way into the training data, like anything else on the internet.

There was a HN post that tested this hypothesis a few weeks ago: https://news.ycombinator.com/item?id=49010129

What is this supposed to mean exactly? Do you think there are developers at OpenAI or Anthropic whose job is to train these state of the art models to draw pelicans riding bicycles? Like how exactly do you expect them to be doing that anyways? Hiring graphic artists to create SVGs of bike riding pelicans and feeding thousands of them into the model's training set?

It stopped being a valuable benchmark proxy quite a few model versions ago. Simon knows it, so is everyone who’s serious about it

Treat it like a bit as is

For the moment the best place to see them all is to browse the tag on my blog - 140 posts now! https://simonwillison.net/tags/pelican-riding-a-bicycle/

I'm running out of excuses not to build a proper comparison site though. Maybe I'll have Astra do rhat.

This tracks with what OpenAI has been saying about Astra: that it tends to do things in a certain way and it’s up to you to prompt it to change its style.

Astra is the only model that correctly depicts occlusion of crank and leg, although interestingly, at max and medium levels (not in between).

Haiku the GOAT

I have a feeling they've been doing that for a while now, even if unintentional, as it's such a well known benchmark

But pelicans are almost never accompanied by helmets, so it cancels out.

> Biking is almost always accompanied by helmets

Isnt Netherlands the leader in bike riders and they dont wear helmets.

At least one reasoning trace I saw considered a helmet and discarded the idea because it was worried about obscuring some detail

Me toooo, for all my personal projects I need to pay for the tokens~ At workplace I use sol because I don't need to pay

Mine as well, it’s fast and keeps me in flow; and cheap!

The price is probably coming from when Luna was first released. OpenAI slashed the price by 80% at the end of July.

I thought all the designers have been using vector graphics for a long time now.

That photo looks pretty hilariously stupid, so this appears to be more of a first toe dip rather than some indication we can one-shot a previously difficult process.

Do you prefer the fable one?

It's more "correct" but looks a lot worse in my opinion:

https://aibenchy.com/compare/openai-gpt-6-astra-high/google-...

Yeah, that's confusing, the score is for the entire benchmark, not for SVG generation only.

Good point about the mouths, I just noticed, lol

Imo, it's still better than most models, I personally like the stylized perspective.

You can view here all generations for all models: https://aibenchy.com/showcase/

I've replaced "Score" there with model ranking, to reduce confusion, thanks for the feedback!

They haven’t activated Astra for me yet, I have two resets now. I’ve been using the opportunity to test out how good 5.6 Sol is at computer use asking it to generate stuff in Blender which has been… interesting

Edit: nevermind it JUST gave me a notification to use it!

That's a pretty genius internal incentive to move fast.

> They gave out bankable resets for every day people on pro plans didn't get Astra.

Yeah, when I saw that Tweet I knew the person was saying it because they knew it'll be available within 24h.

WTHIT?