Remix.run Logo
▲ jsnell 2 hours ago

> - Why are they stacking Fable xHigh against Carbon Med?

Because on that specific benchmark (FrontierCode 2.0), Medium is Carbon's best effort level and xhigh is Fable 5.2's best effort level. You can see that on the graph your next point is about.

And it's just that one benchmark that's at medium. "All scores show each model’s highest-available effort unless otherwise noted".

> - How did they produce such a bizarre effort-intel curve for Carbon that goes against every available benchmark?

The graph is labeled as being about FrontierCode 2.0. Those appear to be Carbon's scores on the official FrontierCode 2.0 leaderboard: https://cognition.com/blog/frontier-code-2.0

> - Why did they leave Carbon-- positioned primarily as a cyber model-- off of the single cyber benchmark comparison?

> - Why do they switch to comparing against Argon (rather than carbon) for robodojo?

Because no numbers are available for Carbon for those benchmarks, and the model isn't publicly available so they couldn't run the benchmarks themselves?

And from your followup:

> where's Astra on that CVE chart

To run Astra on a vulnerability finding eval, you'd need to be part of OpenAI's verified cybersecurity program. Otherwise the results would be tainted by refusals. Do you expect that OpenAI would grant such access in this case?

▲breakwaterlabs 2 hours ago | parent | next [-]

> Because on that specific benchmark (FrontierCode 2.0), Medium is Carbon's best effort level

I am referring to the chart that cross-compares multiple benchmarks, which Artificial Analysis clearly shows Carbon-high as superior on. Further, AA's FrontierCode chart places Carbon-high as having a higher score on Frontier 2.0 than Carbon-med (https://artificialanalysis.ai/evaluations/frontiercode-2-0?m...)

Where it gets very interesting is when you look at the Cognition "FrontierCode" publish date (2026.10.27), and the revision as of 10.28 in web archive (https://web.archive.org/web/20261028175237/https://cognition...). This revision has a VERY different Carbon effort / intelligence curve where high beats medium.

It is only with the latest revision, as of today, that all of a sudden Carbon's benchmark curve goes cone-shaped. Curious, indeed....

I'd encourage you to take a look at a third-party's take on this and see whether Anthropic's self-serving report looks "on the level" after doing so: https://artificialanalysis.ai/?models=claude-fable-5-2-high%...

▲phoghed 2 hours ago | parent | prev [-]

> Do you expect that OpenAI would grant such access in this case?I

Do you expect that nobody at Anthropic has such access? Trillion dollar company can’t manage this basic level of opposition research?