How credible is this benchmark? does it correlated with others real world experience?

bfeynman 2 hours ago | parent | next [-]

Given it was made by cognition (team behind devin flop) who now just got to wait out until claude and gpt5 basically do all of the work for them - not very. When you read about it, the framework is highly subjective. Which very quickly becomes a problem because its based on heuristics that probably change a bunch with a better code model.

	▲	vanuatu 2 hours ago \| parent [-]
		the subjective framework is exactly why its good prior bms relied mostly on unit tests or synthetic judges which are easily benchmaxxed, which leads to nobody trusting benchmarks we need people manually checking the data for good code quality

▲

vanuatu 2 hours ago | parent | prev | next [-]

i worked on one of the benchmarks typically found in new model releases

this benchmark looks very good from the methodology. a cog researcher checking the data themselves is very high signal (not scaleable so don't take the benchmark as gospel, but directionally good)

▲

Catloafdev 2 hours ago | parent | prev | next [-]

It's a relatively new benchmark but from what I can tell it has serious cred behind it. I assume it will be picked up as part of the standard suite of CS-related benchmarks soon enough.

▲

schipperai an hour ago | parent | prev | next [-]

Cognition did well in documenting their approach [1].

TL;DR - they worked with OSS project maintainers to build tasks. They score models based on whether a PR is mergeable. All tasks are graded by a human researcher. SoTA models have hill-climbing to do which raises the bar and inspires confidence. I'd say it's legit.

[1]: https://x.com/cognition/status/2064061031912288715

▲

emp17344 2 hours ago | parent | prev [-]

Seems like it literally popped up yesterday with the express purpose of building hype for this release.

▲

swyx an hour ago | parent | next [-]

team member here - we had been working on frontiercode for ~6-7months. timing just lined up

▲

osti an hour ago | parent | prev | next [-]

And notable absence of DeepSWE benchmark where they do badly, but somehow a benchmark that was published yesterday is in this announcement.

▲

vanuatu 2 hours ago | parent | prev | next [-]

i doubt it, cog wants coding agents to be better because it directly improves their product

they aren't married to a particular lab, most of their usage is their in house model i believe

▲

anthonypasq 2 hours ago | parent | prev [-]

what incentive does Cognition have for doing this? seems like complete nonsense speculation on your part.

▲

bel8 2 hours ago | parent [-]

With billions/trillions of dollars floating around, is it hard to imagine benchmarks could be biased?

I think it's safe to assume everything AI related is heavily biased until proven otherwise. Just like in pharma.

	▲	camdenreslink an hour ago \| parent \| next [-]
		People game benchmarks for fake internet points to get their favorite web framework to the top of the list. I'm pretty sure they will do it for billions of dollars.
	▲	anthonypasq 34 minutes ago \| parent \| prev [-]
		you didnt answer my question. Why would cognition be biased towards making anthropic look good?