Agree. The pelican benchmark was interesting a year ago when most models struggled and a good pelican indicated an unusually capable model. Now it’s saturated and uninteresting.

A good new benchmark should have awful performance to start and there should be a lot of headroom for improvement. This benchmark is also intentionally difficult and requires the LLM to develop the animation through spatial reasoning and first principals rather than existing video generation pipelines. Similar to how SVG generation was out of distribution for most models a year ago.

Aren't they still bad at understanding how bicycle frame works? Especially the steering part?

But it's another benchmark on how good models are at generating intensely average, unwanted things with unthinking design. Just scaled up.

Bad? It has a charming style. I would watch the whole book if it was made like this.

The product you've built the animation for is only for macOS, which is a pity cause it has a way wider appeal than macOS market share is. You have low-hanging fruits right there, don't miss them.

I am interested, I'd love to see! I'm waiting for the day my dad's self published books become self produced movies!

Taking a single paragraph of literary text, which is abstract and ambiguous, and converting it into a 3D animation requires an enormous amount of implicit knowledge about spatial relationships, intuitive physics, everyday objects, and so forth. Not to mention the mathematics of 3D transformations and computer graphics more generally. Saying that it's indicative of no more than three.js coding ability is absurd.

I don't find three.js models/animations as indicative of anything other than the model's ability to write three.js code.

This weekend I've been converting a game from three.js to ogl.js in order to see if I can optimise the time-to-interactive loading time. I took the three.js driven page weight from about 600KB (500KB being three.js) to about 50KB, and reduced the loading time from multiple seconds on a 4G mobile connection to around 0.5s.

This has mostly been a combination of Opus 5 and Sonnet 5 in Claude Code. It very clearly has a good grasp of WebGl, and of what impacts page loading times and rendering speed. It was able to drive Claude Code's integrated browser to measure the impact of changes, and as I spiked out a test of ogl.js it could test the differences changes made.

It's not the best game (https://tinyslots.ooer.com) but that's on me. As an exercise in building 3D in a browser, and in page speed optimization, with Claude models I am really impressed.

Don’t worry. I’m sure they’re not training it to be good at things like writing database backends, financial services, logistics systems, user interfaces, or anything of economic value. As long as you’re not working on three.js specifically, I’m sure Anthropic isn’t making any progress you should be worried about.

It’s a fun idea to re-animate dead google products by feeding product videos to an AI

i remember this failing, but this product looks pretty useful in 2026

which to be fair, happens just a few paragraphs later :) I first watched without sound and thought "oh that's the birthday speech disappearance"

To a certain extent to what end though,

On the one hand yes, almost every task I work at now is one off one of scripts I throw away.

The question is - where does the software the spec or the “code”.

A really complex game will probably always be token heavy. At least for the next few years code is still not free.

But certain software is just iterative by design. If we mean we regenerate all the for loops of a game from scratch, sure but I think “code” Is really more spec then implementation, and we’ll want to continue building things through iteration.

And even on the for loop point - Do you really want to spend millions of tokens rewriting a game every time you need to make balance changes?

I'm sure this will become the standard. And plastic is an excellent analogy. Maybe we can take it a step further and compare it to on-demand 3D printing.

Why would anyone still use off-the-shelf software when they can have a system that has access to all data, can transform it into any form, and can export it in any format?

After years of thinking that I needed to develop a decent movie management system for my own films or a columnar browser for large CSV files, Claude and Qwen each delivered exactly what I needed in just a day.

Yes. It's fantastic for one-off tasks.

Does this make sense with economics of software though? Throwaway products compete with more durable versions of the same product because there is a cost per unit produced that can be minimized by using cheaper materials or production processes that cut corners. With software there is no cost per unit. There might be a market for one-off software that serves a very specific purpose where throwaway software can compete with adapting more carefully engineered software to that purpose, but I am not convinced there is a lot value in this market.

> Just like cheap plastics and improved processes has enabled us to rapidly manufacture anything we want for a very low price

Have they? Most of the world production is tied down to expensive factories and machines. Yes, we have more products, but that the result of the global trade, which is a very complex system.

> Produce it cheaply and if it breaks throws it away and reproduce it.

I don't know why everyone would ever wants this. It's been parroted since forever, but the true usefulness of software is to be able to build it once and runs it indefinitely. If some edge case occurs, I fix it. Which is way cheaper than rebuilding the whole thing. The goal is to have something like OpenBSD's ed[0] or dmesg[1], which you only touch every few years or so

[0] https://github.com/openbsd/src/commits/master/bin/ed

[1] https://github.com/openbsd/src/commits/master/sbin/dmesg/dme...

I’d love to see this benchmark using Blender. Asking a model to animate a scene in Three.js is a square peg/round hole; it doesn’t convey much when the model can’t get it to fit. With Blender the human expert ceiling for this task has been proven to be very high

Coding, graphics, all seem to have very defined background data for an LLM to base their decisions on, test, and even when I correct it we're all on the same page. Seems like there's lots of room for AI productivity there.

What I find funny is that approximate to computer graphics are video games. When I ask AI about a decision available to me in a video game AI completely fails, OFTEN. I assume all the forums and changes made to a game over time might be quite confusing for AI. But I've also seen it completely make up characters and decisions and weapons and so on about some very clearly defined games and paths. It's an interesting dynamic.

"Draw a pelican on a bicycle" is not a serious benchmark.

"Draw an animation of this long ass scene from a movie, and only call me when everything works e2e" can be.

They can't even run a vending machine. I don't want to do a shallow dismissal, but I think there's a gulf between my understanding and yours. I hope it's me so I learn something.

I think LLMs will be excellent glue of "find the right function/button and run/push it" but design without constraints and they just explode immediately

> IMO, the area where AI is going to be most useful over the next couple years is in developing manufacturing processes top to bottom.

Why do you say this? I have some experience of manufacturing processes and am not seeing where AI would be useful other than to drive the robots which we can already do quite well without AI (see all the dark/lights-out factories that already exist).

Where do you see it being useful? An example would be nice.

I think embodied AI or autonomous experimentation will need to make a lot more progress before that kind of thing is possible.

Asking AI to design real world objects doesn't work very well because all of its tests involve proxies and thus miss things that are glaringly obvious when the object is actually built.

You think the bottleneck in creating a sneaker factory is not having an LLM to tell you what machines to buy?

Yes but the second order effect of this is that the cost of the tooling goes up since it is now the bottleneck, and therefore the shoemakers that survive do it off of technical complexity, branding, and regulatory capture.

> "design me a sneaker and all the equipment to manufacture it autonomously"

We already have sneaker designs and the equipment to manufacture them. Whatever it spits out is going to be, at best, a mediocre clone of something that already exists. What exactly is the point?

Gotta love when a techbro just says some complete nonsense like this with total confidence

I’ll never not be amazed that we can type some words and get those results back out.

However I do agree that the results are much worse when you try to use them. They look great in screenshots and video clips which makes them perfect for content farmers.

All of the LLM generated games I’ve played have been really bad to play, though. I even tried my hand at a simple game, thinking I could iterate on it with prompts to fix some rough edges. After the initial productivity burst every change turned into a slog of tokens with one thing changing and something else breaking it. I would try to use my remaining weekly token budget across Anthropic and OpenAI to refine it at the end of every week but after a couple weeks it felt like I wouldn’t be getting anywhere without scrapping it and going back to having the LLM build it one step at a time with my careful instruction.

Which, in retrospect, is the only way I can get usable output of an LLM for anything complicated, so it’s not surprising. It’s a fun reality check project though.

LLMs are bad at creative work and I don’t see them improving any time soon. Try asking an agent to write a story about raccoons. It will almost certainly involve either stealing food or raiding trash with a 50% chance of having a character named Pip. If a location is mentioned, it’ll be Elm Street.

I guess this is the average story and, similarly, the average game is boring and predictable

We're all just impressed that it's even possible. No one is saying these are fun.

I guess the next question is if they can be made fun without too much additional work with a human guiding the AI.

LLMs currently have 0 imagination, I have strong doubts that will be solved any time soon even if they continue to become superhuman at everything else. They just have the creative instincts of a 50 year old accountant.

Right, but the new phenomena is that we've lost a signal.

Used to be if a game looked that good it probably had time spent on the game part too.

It’s called “demo porn” and it’s really obnoxious and borderline offensive to people that take the craft and art form of video games seriously.

If I see one more “one shot MMO” where you just walk around and do absolutey nothing or another menu slop idle battler or rogulike deck builder I’m going to go Postal in Minecraft.

It's the flashy game/movie trailer equivalent for LLMs. Completely unrelated to the real experience, but good for marketing.

Unfortunately the Phaser framework has decided to go all-in on this route, and it's really disappointing.

These one-shot products aren't games. They're barely even demos. I don't even know what to call them. For a mature framework like Phaser to sell-out like this and create a vibecoded platform for vibecoded games is shocking.

It's blockchain for gaming all over again, putting the cart before the horse.

you're tired because you lack imagination.

Agreed that software isn't going to fix our institutions. But we need 0 new software? AlphaFold won its creators a Nobel prize in chemistry and solved a research problem that feeds into every area of biology, medical research, and drug discovery. We still have an untold number of unsolved problems related to human health, food production, energy production, infrastructure, transportation, education... The list is practically infinite.

Beautifully put. Software is ok. It's not going to fix our broken world. If AI could at least find cure for some illnesses that would be great. Or decrease social injustice and help with global warming. But it's likely going to do the exact opposite.

Karpathy works for Anthropic so he obviously has full access to everything, and doesn’t have to pay for token burn either.

> I really dislike this AI programming thing of “Mr LLM, go slam your face into the problem until there’s no problem left, then call me back”.

Not difficult to see why the employees of AI firms are thrilled with it though, eh?

> I don’t think I really have a point to make here, other than it just feels like someone’s released a bunch of carnival bumper cars onto the highways.

I like the analogy.

I guess this is why we got this before cars sold without steering wheels: literal guardrails on the literal roads are somewhat more expensive, especially for the people who keep bouncing off them on the way to their destination.

Also, where the guardrails are absent: oh look, felonies. https://www.google.com/search?q=ai+hacks+company&tbm=nws

Like hitting pinball bumpers, except the high score in the end is the bill you pay the token provider.

I know this is a plug but I thought of it, too! I hope this is the next benchmark they saturate.

I like where Karpathy is going; I had the same thoughts about LLM generated slop scenery. I just want some variety of scenery for the goblins to get massacred in in whatever fantasy slop game I play.

He makes a very good point here about llms or lrms not being good at taking video as inputs, and i think he's signaling his intent to help change that. He's pointing at an open problem. This isn't a carefully structured blog post, it's just a tweet.

> 8 months ago, he was (very reasonably) claiming that reliable agents are at least a decade away, but this now goes against the interest of his employer, so the narrative has been changed.

Or maybe over these 8 months agents improved a lot? You know, few years ago many AI experts predicted that things we are routinely doing now with AI are decades away. I mean how can you look at this post and not be impressed? It's insane what AI is currently capable of.

What narrative has changed ? I just see an experiment with a very perfectible result and some reflections about what capacity are currently missing to have better results.

I'd like to see tests of things the current AIs are bad at, like drive a car. Or maybe take instruction to complete some novel activity to test how well they can learn.

Can someone please translate that expression? What would that mean?

Concerning

> wasting money

Testing, assessing, tasting...

We also do it when we build other things - this is just a different scale.

Do you think this was supposed to be art? Wow.

I don't really get the desire for hyper-personalized entertainment. People are very good at pointing out things they dislike, but not very good at coming up with how to fix them (common wisdom in game design).

On top of that, a decent chunk of the joy of entertainment is the social aspect.

If that becomes a thing, that will be fun 2 weeks, then become a gimmick only a small niche will be using. What’s the point in having a hyper personalized entertainment? People want to experience the games, movies, tv shows, books others created and are also experiencing

I very much doubt that. We now have nearly-perfect AI image generation and it hasn't really changed the nature of human expression. I don't see my friends getting wildly creative. I mostly see it used for spammy blogs, spammy books, and cringeworthy corporate marketing - basically, a negative signal, rather than "ooh, AI image, I'm in for a treat". Is your experience different? If not, what changes with moving images?

Ultimately, most people don't have ideas for the kinds of personalized entertainment they want, and they don't want to be in charge of content production (even if you have an LLM do most of the work). I don't doubt that there are niches for it, especially stuff like porn, and I'm sure that pros (game studios, film studios) will leverage AI more and more, but I suspect that most of us will just want to sit on the couch, watch Spiderman XVIII, and then be able to talk about that shared Spiderman XVIII experience with all our friends.

Judging by the Seedance 2.5 demos today, I’d say it’s not that many orders of magnitude away now.

I kind of like a sense of community even if it means losing out on personalisation.

Amazon recently kiboshed a new Stargate television series. The fan community was quite heartbroken because the Amazon had involved some of the writers from the original television series and apparently one of the reasons that they cancelled the show before it made it to production was that the felt that the proposed story would appeal too much to the fans and not new people.

The fans were real heart broken about this but I think you're right on where this is going. We're not going to be seeing the dominance of centrally produced content like this for much longer, like sure, I think there will be big blockbusters will stick around, but I think the day is coming where media becomes a choose your own adventure sort of scenario.

It'll be interesting to see where this scales to. there will definitely be some amazing solo projects but we'll also see the like 4 player co-op version of productions and then the larger mine-craft server 'Minas Tirith' scale ambitious projects that involve a few dozen people. And of course passive consumers will remain a thing, or people who just provide some suggestions or nudges for what they'd like to see others make.

But I don't think it'll be dominated by big companies like Disney, Netflix, Amazon or Paramount.

I'm sure a lot of them will suck but it'll be neat to see the inevitable Seinfield - Star Trek Voyager cross over episodes.

Elaine and B'Elanna Torres feud after a transporter accident leaves the crew stranded the delta quadrant. Jerry attempts to date 7of9 but is rebuffed as she finds Kramer's quirky bluntness more relatable. George panics after someone compares him to Neelix.