| ▲ | simonw 5 hours ago |
| llm -m meta-ai/muse-spark-1.3 "Generate an SVG of a pelican riding a bicycle"
https://tools.simonwillison.net/markdown-svg-renderer?url=ht...4.2266 cents, 38 seconds. For comparison here's Muse Spark 1.2, which animated it without me asking it to: https://tools.simonwillison.net/markdown-svg-renderer?url=ht... The 1.3 one is definitely better - better bicycle frame, better wing, better pelican hat. UPDATE: Here's another one with five pelicans for each of the five Muse Spark 1.3 reasoning levels: https://tools.simonwillison.net/markdown-svg-renderer?url=ht... The most expensive was reasoning level xhigh - 7.5 cents, 1m34s. And I ran five pelicans at all reasoning levels for 1.2 as well, here: https://tools.simonwillison.net/markdown-svg-renderer?url=ht... |
|
| ▲ | wewewedxfgdf an hour ago | parent | next [-] |
| I was interviewed for a job as a software developer last week and they asked me to draw a picture of a pelican riding a bicycle. Aced it, got the job as a senior software engineer. The interviewers afterwards said "it is SO refreshing to find a software developer who actually knows how to code - never seen such a high performance focused, well built pelican on a bike - you have the skills we need". |
| |
| ▲ | fuddle an hour ago | parent [-] | | I also aced my interview by focussing on pelicancode problems, instead of leetcode problems. | | |
|
|
| ▲ | drusepth 4 hours ago | parent | prev | next [-] |
| Is there a reason these pelicans always have roughly the same composition (side-view, 2d, biking right, flat ground beneath, etc)? I don't see any of that detailed in the prompt, yet they all seem to generate roughly the same image of differing quality. |
| |
| ▲ | vunderba 4 hours ago | parent | next [-] | | The more generic your prompt, the more generic the response. It's a regression to the "mean" of the training data aka GIGO for AI. It's like when you ask your average person off the street to draw a house - it'll almost always be square with a triangle roof, one door, and two windows. In the pelican/bike example, it's probably a bit of a self-perpetuating snowball too. If the earliest examples were bike left-to-right, flat ground, etc. then they are also being scraped up in future LLMs. | | |
| ▲ | werdnapk 36 minutes ago | parent | next [-] | | Well, all the LLMs are being trained on previous pelicans, so they look the same. | |
| ▲ | polyterative 4 hours ago | parent | prev [-] | | as a kid I did them like this. nobody told me to do that. are we all so similar? | | |
| ▲ | Gigachad an hour ago | parent | next [-] | | It's just the simplest most recognizable form of a house. Like how a smiley face is so generic and simplistic but everyone will know what it represents. Just two dots and a line yet it's easily and unambiguously understood to represent a human face and a happy emotion. | |
| ▲ | collabs 4 hours ago | parent | prev | next [-] | | I sincerely believe I've never had a single original thought™ in my whole life. There is this scene in the HBO series Westworld where a "host" says some words in sequence which is shown on a display as she says it. Of course, even me thinking of this scene and connecting it to your comment was not original, someone else clearly had the same programming as me. A medium blog post says > Pair what with me?” — the moment Maeve (a humanoid android) uttered those words in Westworld (Season 1, Episode 6: “The Adversary”), something clicked. Not for the average viewer, but for me, a STEM educator and AI enthusiast who, just weeks earlier, had read Stephen Wolfram’s seminal essay, What Is ChatGPT Doing … and Why Does It Work? | | |
| ▲ | ACCount37 an hour ago | parent [-] | | Westworld is such a time capsule. It's not even that old - but back when it was aired, an AI that can not just string together coherent sentences, but produce coherent reactions in novel, fully unintended contexts, like Maeve was doing there? It was totally a sci-fi premise. Now we have AIs capable of that and more, and no one bats an eye. |
| |
| ▲ | srcreigh 4 hours ago | parent | prev [-] | | The adults brainwashed us https://www.ikea.com/ca/en/p/barndroem-box-beige-70560615/ https://www.ikea.com/ca/en/p/vallaby-rug-green-10548216/ |
|
| |
| ▲ | BeetleB 2 hours ago | parent | prev | next [-] | | Not when rendered via POV-Ray: https://blog.nawaz.org/posts/2025/Oct/pelican-on-a-bike-rayt... I plan to update it with more pelicans from all the models released since. (Spoiler alert: They haven't improved much since then). | | |
| ▲ | xhrpost 2 hours ago | parent [-] | | Wow, I actually had this exact idea. I was specifically curious as to how well a given LLM could understand a DSL that hasn't changed much in a couple decades and doesn't have nearly as many examples to learn from online. Seems like it did alright, all things considered. |
| |
| ▲ | simonw 4 hours ago | parent | prev | next [-] | | It's really interesting, isn't it? They almost always cycle from left to right - but I have had a few which cycle in the other direction. The 2D / flat ground feels reasonable for a SVG, which implies a vector illustration. | | |
| ▲ | m12k 4 hours ago | parent | next [-] | | It's my impression that it's common in western culture, where text is read left to right, and timelines are visualized as going from left to right, to also animate things going from left to right, since westerners thus have an instinct that "right = forward", so it "feels right" (familiar). I wonder to which degree this is reflected in the training data? And if you'd be more likely to get left-facing pelicans if you prompted it in Hebrew, Arabic or another right-to-left language? | |
| ▲ | johntb86 4 hours ago | parent | prev | next [-] | | Someone studied this (among other thigns): https://dylancastillo.co/posts/pelicanmaxxing.html . Pelicans on bikes always face right in this test, but other animals on other transportation methods sometimes face left. | |
| ▲ | piker 4 hours ago | parent | prev [-] | | I was going to ask the exact same question earlier but deleted it after thinking “I’m sure Simon has done some sort of discussion on this.” Since it does seem novel to you, too, it would be really interesting to read more about this phenomenon. |
| |
| ▲ | porphyra 2 hours ago | parent | prev | next [-] | | The canonical view of a bicycle is facing right. Usually, people want to draw/photograph/depict the side of the bicycle with the running gear, which is on the right side of the frame for historical reasons. | |
| ▲ | __MatrixMan__ an hour ago | parent | prev | next [-] | | The thing that distinguishes pelicans from other birds does so most strongly in profile. If you're looking straight at one, the throat pouch would be hidden by the beak. I bet if it instead had something to do with black widow spiders we'd find that we're most often looking at the bottom of the spider's abdomen, regardless of whatever non-spider-like activity is supplied. | |
| ▲ | postalcoder 4 hours ago | parent | prev | next [-] | | Search Google Images for "bicycle". Almost all bicycle product shots are staged the same way: side view, going left-to-right. It makes sense to me that given that skew in the training data, the model grounds itself in the bicycle. | | |
| ▲ | daemonologist 4 hours ago | parent | next [-] | | and furthermore, this is because the drivetrain is ~always on the right side of the bike - if you want to inspect or admire a bicycle you look at the right side, as you might look under the hood of a car. (Why the drivetrain is on the right, I don't know. But most bike parts follow open standards so it's quite entrenched.) | | |
| ▲ | labcomputer an hour ago | parent | next [-] | | I can’t tell you why it’s always on the right, but it’s always on the same side because of network effects. Bicycle frames are not fully symmetric left-right because you need things like a mount point for the derailleur hanger, and optionally affordances to keep the chain off the stays when the wheel is removed. Those things have to be on the same side as the chain. Bikes designed for disc brakes additionally need a mount point for the brake caliper on the opposite side from the chain. Additionally, rear wheels are not symmetric: the spokes on the chain side connect to the hub closer to the plane of the rim. That is, they are more perpendicular to the wheel’s rotational axis than spokes on the opposite side (which is why you should always mount a single pannier on the chain side). This asymmetry is to provide space for the gears. So once the industry decided to put the chain on the ride, you can’t very well make a group set designed for a left chain if you want it to work on the vast majority of frames. | |
| ▲ | kibae 4 hours ago | parent | prev | next [-] | | Since most languages read from left to right, rightward movement tends to read as forward progression. So when showing a bicycle in side profile, having it face right feels more naturally like it’s moving forward. | |
| ▲ | georgemcbay 4 hours ago | parent | prev [-] | | > and furthermore, this is because the drivetrain is ~always on the right side of the bike While I'm sure this factors into things for advertisements for bike components, there is also just a general preference that westerners have for left-to-right motion. Not just in bike ads, but all ads with (or suggesting) movement. And also not just ads, but movies where directors believe left-to-right motion is associated with progression and right-to-left motion is regressive. |
| |
| ▲ | threetonesun 4 hours ago | parent | prev [-] | | Product shots yes, people riding them its more like 50/50. Also if you search for a specific bicycle race you'll find more going right to left. |
| |
| ▲ | 4 hours ago | parent | prev | next [-] | | [deleted] | |
| ▲ | optimalsolver 4 hours ago | parent | prev | next [-] | | Sun is missing a few rays and not wearing sunglasses. | |
| ▲ | ModernMech 4 hours ago | parent | prev | next [-] | | Yes, I do a thing where I ask the machine to generate responses in the form of a lizard talking to a cat. The lizard is always a green gecko and the cat is always orange, which I never specify. | |
| ▲ | reaperducer 4 hours ago | parent | prev [-] | | Is there a reason these pelicans always have roughly the same composition Because they're computers. They don't have an imagination and the ability to create things from whole cloth the way humans do. Much like a mother pelican, they regurgitate what they've been fed. |
|
|
| ▲ | andytratt 2 minutes ago | parent | prev | next [-] |
| excellent thread |
|
| ▲ | pavs an hour ago | parent | prev | next [-] |
| FYI, your renderer breaks with error "git api access error 403", rate limiting error from git, when using cloudflare vpn. I am guessing its not super common, but it happens just so you know. |
|
| ▲ | tintor 3 hours ago | parent | prev | next [-] |
| Did any LLM so far draw pelican knees correctly and have them bend in opposite direction from human knees? Knees of many animals bend opposite to humans. Did any LLM draw the front bicycle wheel correctly? ie. center of front wheel slightly AHEAD of steering wheel axis. This is done for bicycle stability. |
| |
|
| ▲ | jonahx 4 hours ago | parent | prev | next [-] |
| If you have a grading rubric, huge points off for adding arms instead of using the wings as arms! |
| |
| ▲ | Fergusonb 4 hours ago | parent [-] | | I think it's hilarious that this detail is enough for me to dismiss looking into the model, but here we are, and it is. |
|
|
| ▲ | hollowturtle 3 hours ago | parent | prev | next [-] |
| Is there any point anymore regarding this svg test? I would not be surprised if in the training they're fine tuned for this task too |
| |
| ▲ | cheesecakegood 11 minutes ago | parent | next [-] | | It also works as extremely effective engagement farming, for lack of a better phrase | |
| ▲ | tintor 3 hours ago | parent | prev | next [-] | | It would be very embarrassing for any lab to benchmaxx the pelican on bicycle svg prompt, since it would be very easy to detect it by varying the prompt. | | |
| ▲ | nojs an hour ago | parent [-] | | The amount of discussion around it means that the test and all the reviews of results, images, approaches etc are implicitly included in training data. It’s not deliberate “benchmaxxing” but things that are discussed a lot online are naturally things that LLMs learn better. |
| |
| ▲ | BeetleB 2 hours ago | parent | prev [-] | | You win this thread's prize: https://news.ycombinator.com/item?id=49538333 |
|
|
| ▲ | drob518 3 hours ago | parent | prev | next [-] |
| Simon, at this point I really wonder if teams aren’t gaming this. You should pick a random animal doing a random thing every time. |
|
| ▲ | ipsum2 2 hours ago | parent | prev | next [-] |
| All of the links show "Error: Gist API returned 403". |
|
| ▲ | leumon 2 hours ago | parent | prev | next [-] |
| next, try: "generate an svg of a human hand". this is a prompt where many models fail imo. |
|
| ▲ | jonplackett 4 hours ago | parent | prev | next [-] |
| Has any ab tried to game this yet and just made the most amazing pelican by hand and always reply with that? |
|
| ▲ | jttnr 4 hours ago | parent | prev | next [-] |
| I wonder, given Simons reputation in AI benchmarking, whether model providers try to train or tweak their models to perform better at drawing bicycles and pelicans? |
|
| ▲ | EugeneOZ 4 hours ago | parent | prev | next [-] |
| Absolutely BRUTAL! :) Thank you for doing this, I love your benchmark the most! |
|
| ▲ | tomrod 4 hours ago | parent | prev | next [-] |
| What does the mean pelican look like at this point? Also 3X token use vs. 1.2 |
| |
|
| ▲ | 0xbadcafebee 3 hours ago | parent | prev | next [-] |
| For all the comments of "I'm sure they're fine-tuning for pelicans": https://dylancastillo.co/posts/pelicanmaxxing.html "Sorry, HN haters, but there’s little evidence that AI labs are pelicanmaxxing.
Or at least they’re not doing it in a plainly obvious manner."
|
|
| ▲ | jmkni 4 hours ago | parent | prev | next [-] |
| lol Definitely an upgrade over 1.2 |
|
| ▲ | NoOneCares44 4 hours ago | parent | prev [-] |
| [flagged] |
| |
| ▲ | aldanor 4 hours ago | parent | next [-] | | Someone did care enough to create a throwaway account to vent here, it seems. | |
| ▲ | tonyhart7 4 hours ago | parent | prev [-] | | how tf you can have green names with negative karma ??? |
|