Academically a - very - interesting article, which confirms my ongoing suspicion of LLM Benchmarks in general ( probably why I like it so much). Another thing that caught my eye is DeepSWE having a bunch of bugs in them. The more I learn about these benchmarks the more sceptical I become. I would expect such a prolific benchmark to have these bugs ironed out by now.

The practical takeaway, however, seems to be an interesting "don't use max mode". The passage about deepseeks weird API behavior dropping to pretty much zero the moment you used high says a lot.

Also, personally, even though Opus is clearly so much ahead of the Open Source models after the changes, I think the adjusted score is still very much competitive. I see other commenters using this as proof to justify why their opinions of the open source models - but a drop from 74% to ~50% is, while a big drop, still about a ~50% pass rate afterwards. That's far from useless for coding, especially since DeepSWE seems to be such a difficult benchmark. It simply seems to more accurately reflect the capabilities of these models.

All in all, I'd love to see more of such articles, and will probably keep an eye on this one - if they continue "benchmarking" these models this way.

▲

i2talics a day ago | parent | next [-]

Who has incentive to improve the quality of the benchmarks?

▲

rat9988 a day ago | parent | next [-]

AI Labs actually. They would love the perfect benchmar representing real usecase. I'd even dare and guess that they don't share much publically because it is a huge competitive advantage.

▲

adastra22 a day ago | parent [-]

> they would love the perfect benchmar

That right there is the problem.

▲

stalfie 20 hours ago | parent | next [-]

It is "a" problem, but honestly not a huge one. Chasing benchmarks is an integral part of AI research, and it is how we have gotten this far, which is pretty damn far. Discovering the ways in which benchmarks fail is in itself an interesting process, and the amount of interesting new benchmarks has grown with the capabilities of the models as a result. The frontier labs might be a bit selective about which benchmarks they publish as marketing, but overall they have so far been reasonably diligent about avoiding outright cheating. Most of the truly benchmaxed models are not on the frontier (eg. see muse spark performance dropping off a cliff in the article).

This is a scenario where the interest of the users and the labs actually align perfectly, since labs that have people actually paying for their product want benchmarks to align as close as possible to real life performance. Otherwise they can't actually easily tell whether the new model they are developing is any good or not, and so risk wasting literal billions of dollars on training runs for a product that people won't end up paying for. A failed major training run is an absolute disaster at the frontier, GPT-4.5 for instance was an almost existential failure for OpenAI at the time.

Case in point, the most popular coding model, Opus 5.5, also just happens to be the only one that avoided the benchmaxing impulse in this article. This is the result of years of Anthropic working very hard to avoid it.

	▲	adastra22 10 hours ago \| parent [-]
		Your last paragraph undermines the first two.

▲

rat9988 21 hours ago | parent | prev [-]

What is the problem?

	▲	RinTohsaka 19 hours ago \| parent [-]
		[flagged]

▲

adastra22 a day ago | parent | prev | next [-]

The people who have no power over the situation: users.

▲

s1artibartfast a day ago | parent [-]

Users always have the choice to ignore benchmarks or run their own

▲

danuker a day ago | parent | next [-]

Exactly. Even a brief one (a single task done on two models) showed me GLM 5.3 Flash was superior to DeepSeek 4.1 Flash.

DS used nearly twice the budget to accomplish the same thing.

Ideally, these tasks should be something you actually need day to day.

	▲	lisplist 4 hours ago \| parent [-]
		I'd rank it DeepSeek 4.1 Flash 1017 > GLM 5.3 Flash > DeepSeek 4.1 Flash (Original) - 1017 is a huge improvement.

▲

adastra22 a day ago | parent | prev [-]

That’s missing the point. Frontier models get shittier in certain ways with each release, due to benchmaxxing, and there is nothing we can do about that.

▲

Bolwin a day ago | parent | prev [-]

The benchmark makers and anyone who relies on benchmarks

	▲	verdverm 17 hours ago \| parent [-]
		depends, are they optimizing for accuracy or something else?

▲

joshonyc a day ago | parent | prev | next [-]

Very much disagree, to me it clearly shows the gap between Open Source and the frontier. You can not do real code work with open source models right now.

Especially since this benchmark is still mostly DeepSWE, and the only reason the open source models are doing decent is because much of the information hasn't changed. The open source models have been benchmaxxed to death.

	▲	girvo a day ago \| parent \| next [-]
		> Very much disagree, to me it clearly shows the gap between Open Source and the frontier. Yes, this is true. > You can not do real code work with open source models right now. This is false, or you're using "real" as a loophole. You can absolutely do real coding work with open source models, right now, right this second. I'm doing so today using Qwen 3.8 Flash Next running locally, and GLM 5.3 Flash hosted. We could do real work with Opus 4.5, 4.6. These models (for coding) are more than good enough to do work that Opus 4.6 could do, and in some ways are superior.
	▲	lisplist a day ago \| parent \| prev \| next [-]
		Even if frontier models are much stronger than open weights, strong disagree on this. You very much can get real work done using open weights models.
	▲	codygman 7 hours ago \| parent \| prev \| next [-]
		People who use mythos for everything also say you cant get real work done with opus. You just adapted to niceties of your chosen model.
	▲	verdverm 17 hours ago \| parent \| prev \| next [-]
		> You can not do real code work with open source models right now. I only use open weights and get plenty of real work done. Turns out most real tasks are not that hard, rather they are tedious.
	▲	gabucino a day ago \| parent \| prev [-]
		[dead]

▲

onlyrealcuzzo a day ago | parent | prev [-]

This DEFINITELY squares with my anecdata that Gemini 3.8 Flash is not even remotely close to Opus 5.5 on the types of problems DeepSWE CLAIMS to be benchmarking.

Astra does seem close.

I'd be curious to see Argon on there, since my experience is that Argon is also significantly behind Opus 5.5 and Astra despite the benchmarks (obviously much closer than Flash, though).

I've been meaning to take a closer look at what's behind the curtain of DeepSWE - since I built what I thought should be the easiest possible coding benchmark - and the models I tested it on scored roughly the same as they do on DeepSWE... Which is supposed to be hard...

Something seems amiss.

	▲	jtwaleson a day ago \| parent [-]
		We recently benchmarked our AI code review agent (just pi with some custom prompts based on info from our context layer). Previously we used Gemini flash 3.8 as it was capable and affordable. However, haiku 5.5 just blows it out of the water. It's more than 10x cheaper, 2x faster and finds 3x as many real issues.