new | show | ask | jobs Github

btown 5 hours ago

Thank you for continuing to maintain the only benchmarking system that matters!

Context for the unaware: https://simonwillison.net/tags/pelican-riding-a-bicycle/

▲

l_eo 2 hours ago | parent | next [-]

They will start to max this benchmark as well at some point.

▲

ljm an hour ago | parent [-]

It's not a benchmark though, right? Because there's no control group or reference.

It's just an experiment on how different models interpret a vague prompt. "Generate an SVG of a pelican riding a bicycle" is loaded with ambiguity. It's practically designed to generate 'interesting' results because the prompt is not specific.

It also happens to be an example of the least practical way to engage with an LLM. It's no more capable of reading your mind than anyone or anything else.

I argue that, in the service of AI, there is a lot of flexibility being created around the scientific method.

	▲	tylervigen an hour ago \| parent \| next [-]
		For 2026 SOTA models I think that is fair. For the last generation of models, and for today's flash/mini models, I think there is still a not-unreasonable binary question ("is this a pelican on a bicycle?") that you can answer by just looking at the result: https://simonwillison.net/2024/Oct/25/pelicans-on-a-bicycle/
	▲	interstice an hour ago \| parent \| prev [-]
		So if it can generate exactly what you had in mind based presumably on the most subtle of cues like your personal quirks from a few sentences that could be _terrifying_, right?

▲

gabiruh 4 hours ago | parent | prev [-]

It's interesting how some features, such as green grass, a blue sky, clouds, and the sun, are ubiquitous among all of these models' responses.

	▲	derefr 2 hours ago \| parent \| next [-]
		It is odd, yeah. I'm guessing both humans and LLMs would tend to get the "vibe" from the pelican task, that they're essentially being asked to create something like a child's crayon drawing. And that "vibe" then brings with it associations with all the types of things children might normally include in a drawing.
	▲	btown 2 hours ago \| parent \| prev [-]
		If you were a pelican, wouldn't you want to go cycling on a sunny day? Do electric pelicans dream of touching electric grass?