new | show | ask | jobs Github

breakwaterlabs 2 hours ago

- Why are they stacking Fable xHigh against Carbon Med?

- How did they produce such a bizarre effort-intel curve for Carbon that goes against every available benchmark?

- Why did they leave Carbon-- positioned primarily as a cyber model-- off of the single cyber benchmark comparison?

- Why do they switch to comparing against Argon (rather than carbon) for robodojo?

▲

jsnell an hour ago | parent | next [-]

> - Why are they stacking Fable xHigh against Carbon Med?

Because on that specific benchmark (FrontierCode 2.0), Medium is Carbon's best effort level and xhigh is Fable 5.2's best effort level. You can see that on the graph your next point is about.

And it's just that one benchmark that's at medium. "All scores show each model’s highest-available effort unless otherwise noted".

> - How did they produce such a bizarre effort-intel curve for Carbon that goes against every available benchmark?

The graph is labeled as being about FrontierCode 2.0. Those appear to be Carbon's scores on the official FrontierCode 2.0 leaderboard: https://cognition.com/blog/frontier-code-2.0

> - Why did they leave Carbon-- positioned primarily as a cyber model-- off of the single cyber benchmark comparison?

> - Why do they switch to comparing against Argon (rather than carbon) for robodojo?

Because no numbers are available for Carbon for those benchmarks, and the model isn't publicly available so they couldn't run the benchmarks themselves?

And from your followup:

> where's Astra on that CVE chart

To run Astra on a vulnerability finding eval, you'd need to be part of OpenAI's verified cybersecurity program. Otherwise the results would be tainted by refusals. Do you expect that OpenAI would grant such access in this case?

	▲	breakwaterlabs an hour ago \| parent \| next [-]
		> Because on that specific benchmark (FrontierCode 2.0), Medium is Carbon's best effort level I am referring to the chart that cross-compares multiple benchmarks, which Artificial Analysis clearly shows Carbon-high as superior on. Further, AA's FrontierCode chart places Carbon-high as having a higher score on Frontier 2.0 than Carbon-med (https://artificialanalysis.ai/evaluations/frontiercode-2-0?m...) Where it gets very interesting is when you look at the Cognition "FrontierCode" publish date (2026.10.27), and the revision as of 10.28 in web archive (https://web.archive.org/web/20261028175237/https://cognition...). This revision has a VERY different Carbon effort / intelligence curve where high beats medium. It is only with the latest revision, as of today, that all of a sudden Carbon's benchmark curve goes cone-shaped. Curious, indeed.... I'd encourage you to take a look at a third-party's take on this and see whether Anthropic's self-serving report looks "on the level" after doing so: https://artificialanalysis.ai/?models=claude-fable-5-2-high%...
	▲	phoghed an hour ago \| parent \| prev [-]
		> Do you expect that OpenAI would grant such access in this case?I Do you expect that nobody at Anthropic has such access? Trillion dollar company can’t manage this basic level of opposition research?

▲

polski-g an hour ago | parent | prev | next [-]

Because Carbon is not out, nobody can run tests on it.

	▲	scrollop an hour ago \| parent [-]
		Arena.ai released carbon tests 12 days ago - the only service that I have seen on yt that gained access. And you can compare to other SOTA models in their video. https://www.youtube.com/watch?v=NO2I3ueVQrk And they have one on fable 5.2 that was just released, yay https://www.youtube.com/watch?v=QwaU4yU0PdQ

▲

marshalla 2 hours ago | parent | prev | next [-]

yes. this.

▲

breakwaterlabs 2 hours ago | parent | prev [-]

Just to follow-up-- artificial Analysis shows no justification for that Carbon effort curve, but it does show Carbon on medium smashing Fable on things like Terminal-Bench and AA-Omniscience v1.1 Non-Hallucination Rate.

But I suppose the reality is rather obvious when you note the funny games they're playing with the graphs, and the things they won't talk about (where's Astra on that CVE chart?)