Why would you get emotional over a model? They got you that good?

I do think it’s the wizard not the wand at this point given a decent model. These benchmarks don’t have the wizard.

Otherwise I wouldn’t see others in the exact same codebase struggle and underutilize agents while others thrive using the exact same ones.

Sol failing mostly on “unverified assumptions” and rarely hitting “integration errors” seems about right to me. I think Sol is second only to Astra (and miles ahead of even Fable) in architecting & engineering the right implementation — but only if you are extremely specific and provide tight guidelines and guardrails. If you give it a one-liner… you’re going to have a bad (SHA-256-hash-verified) time.

Here's the list:

1 Fable 5.1 38.8%

2 GPT-6 Astra 33.8%

3 Gemini 3.8 Flash 31.2%

4 GLM 5.3 28.8%

5 Grok 4.6 23.8%

5 Muse Spark 1.3 23.8%

7 Kimi K3 18.8%

8 GPT-5.6 Sol 16.2%

See number 4, 5 and 7.

3 of those are already in there.

Do you find Fable significantly better than Opus at avoiding-overengineering? All of my recent testing of Anthropic models seems like they're tuned-to-hell to (a) be much slower than they need to be (running tests over and over during the loop vs at the end, say, even if those tests take a few minutes a pop) and (b) doing exactly that sort of "built a lot of fancy enterprisey feature-adjacent 'stuff'" even before nailing the actual feature. Sol and Terra both have some of the latter but they seem to do the actual work a fair bit faster (this may be a usage-based-priority-tier/rate-limit thing though) which helps offset it.

I think the bigco folks saw all the "it wrote all this code but the tests didn't pass" or "it wrote the feature but it's super brittle" and tuned the newer model+harness combinations incredibly aggressively to try to turn a lazy prompt into "median Enterprise Architecture design suggestions" to bring up the baseline, but in a way that slows you down if you don't want that.

I'm not on big enough subscriptions to want to burn a lot time just evaluating Fable/Astra comparatively until they're cheaper, heh. I can steer any of the cheaper ones just fine anyway.

Anecdotally, +1. I’d also say this benchmark matches my experiences and how much I trust the model output

Astra constantly does this for me. It goes 90% of the way with some task but then skips the most important part. Then when told to please fix that and do it properly, it suddenly goes down a rabbit hole for 6h and fixes scenarios that aren't even relevant. It's awful at assessing what is important to do and what not, and where to ask for permission and where not.

> How does that work?

My gut feeling is that any serious real-world company with a proprietary codebase worth looking at would not be handing out the crown jewels to a third party. License or not.

I don't doubt somebody licensed their codebase to them, I just have my doubts about who the "who" could be.

lot of ads everywhere offering to buy your codebase of real product/star up even it long gone or failed (offer usually price per lines of code). So most likely that they have bunch of abandoned codebases between small and medium sizes and probably also some fake codebases as well.

Terminal bench 4 is good largely because it's recent so it hasn't been benchmaxxed yet. It's more of a sysadmin/devops benchmark than a coding benchmark though, but still a decent proxy.

https://artificialanalysis.ai/evaluations/terminalbench-v4-0

Epoch.ai has a global score and tracks many benchmarks: https://epoch.ai/benchmarks

Nvidia and OpenAI claimed AGI, but you still have a job.

Astra lets me offload entire projects without worrying about individual tasks…

If it builds up history and perceived reliability, this type of thing can be valuable. You're giving up transparency for it being harder to game.

Lots of private benchmarks already exist, where you have to trust the tester (ex Artificial Analysis, Arc-agi).

In theory, as long as all the models are doing the same thing with the same tools, it's at least useful to see how they stack up against each other right now. It might not be great to track progress over time, as it can get benchmaxxed or the underlying resources may become obsolete.

Doesn’t it ultimately have to be this way, to prevent saturation?

I switched from Claude to Codex because Claude just doesn't do what you actually tell it to half the time. It dances around the edges and does busy work without actually tackling a tough problem.

I'm not sure what others are doing that they're getting such different results, but I'll take Codex every day of the week.

They're not measuring speed nor annoyance. It's there on the page

havent tried that model, but it sounds like a potential harness issue. Have you tried it in different harnesses?

Hard disagree. I use 3.8 flash in Antigravity a lot, and thoroughly prefer it to most Pro-class models. It's really fast, and I've had it make crazy progress on compiler-like problems that previous models including Opus simply failed at. On ultra plan you can have it going for hours, and make incremental progress with good prompting for review interrupts. It solved a problem I couldn't solve for weeks in under 6 hours. 10k LOC total. The harness and test suite is key.

Why so brazenly confident? Isn’t it possible that the benchmark is correct, and your experience is correct too, but you haven’t tried all the thousand different modalities of work that programming encompasses and so maybe you don’t actually have standing to judge?