> That’s the problem, these tests can’t tell you how good a model is anymore because it’s trivial for labs to optimise for exactly these tests by the next release.

But they don't prove the claim. Are the models amazing at recreating Minecraft, but the second you swap the word Minecraft out with another game or a custom game, it shits the bed?

That's not what I see. My feed is full of people using Astra to recreate all sorts of games from Diablo to some random idea they came up with, in ridiculously polished detail like animations that would have taken me weeks of iteration in gpt-5.6-sol but it was a single shot by Astra.

> ridiculously polished detail

Can you point me to one of these? The only Astra game I tried was the incredibly janky Mario Kart clone from OpenAI where you could fly by spamming jump

You are falling for selection bias.

For every person using Astra to create a game from some random idea and sharing the impressive result, there are an unknown number who have tried the same thing and gave up in frustration.

I call bullshit