It does vey well at one shotting a PacMan clone, pretty much perfect. https://jonclegg.github.io/pacman-bakeoff/entries/claude-son...

2nd only to Opus 5.5, which is perfect. https://jonclegg.github.io/pacman-bakeoff/entries/claude-opu...

Up until very recently, all models struggled with this.

All results: https://jonclegg.github.io/pacman-bakeoff/

▲

fakedang 2 minutes ago | parent | next [-]

Interesting. Sonnet 5 was horrible, and Opus 5 was unplayable, but both Sonnet 5.5 and Opus 5.5 were about as close to the real thing.

▲

judge2020 2 hours ago | parent | prev | next [-]

Oh, it coded a Pac-Man clone. The clone was so good that I thought it was premade in some way and that Sonnet was going to play PacMan.

▲

thefourthchime 2 hours ago | parent [-]

Yes! The point being that up until yesterday, every model struggled with this, and now they don't.

▲

copperx an hour ago | parent | next [-]

"this" being recreating Pacman specifically, or games?

▲

londons_explore an hour ago | parent [-]

I made ~10 games with opus 5.5 (all multiplayer web games over web sockets).

About half the time it made a playable game in a single short prompt. The other half of the time a few follow-up prompts were needed for refinement (eg. Things like "the blaster weapon is way too powerful, divide it's hit points by 10" or "we need a way to reconnect a player whose network dropped mid round" or "the GPS doesn't work on iOS")

	▲	copperx an hour ago \| parent [-]
		Other models fail at oneshot creation of similar games?

▲

thefourthchime an hour ago | parent | prev [-]

Your welcome!

▲

sixtyj an hour ago | parent | prev | next [-]

I have played few of them and it seems that Opus 5.5 is the first one who really made playable PacMan clone game. On mobile as well.

Could it be because the model was somehow pre-trained? If we compare it with pelicans that are still not-perfect…

▲

sally_glance an hour ago | parent | prev | next [-]

Cool page and benchmark idea! Would be nice if there was some kind of grading the results, maybe on different criteria (aesthetic, implementation complexity, correctness, ...). Of course as a one-shot and greenfield benchmark the results are not indicative for all kinds of usage patterns. But as some sibling said, maybe they can be indicative on some general characteristics (especially since the task is so open-ended).

▲

pyaamb an hour ago | parent | prev | next [-]

Very cool. I'd love to see someone with access to plenty of token$ make something similar for the "Browser Desktop OS" test. That seems like a pretty comprehensive test thats also fun to test just like this!

▲

nicce an hour ago | parent | prev | next [-]

I was able to get similar with Qwen 3.8 27B with one shot. I think this game is too well in the training data.

▲

ilamont 2 hours ago | parent | prev | next [-]

Thank you for doing this. It is very helpful not just for capabilities but also for costs.

▲

russellbeattie 2 hours ago | parent | prev | next [-]

Wow, that "bake off" page is better than any coding benchmark I've seen! You can really sense the strengths and weaknesses of each model/harness combo.

	▲	thefourthchime an hour ago \| parent [-]
		Thanks!

▲

formvoltron 25 minutes ago | parent | prev [-]

oh! How about pengo, dig dug, & defender?