I think it's also telling that they left out the usual hallmarks of the Pareto distribution: GLM 5.2, Qwen 3.7, Minimax M3, and Mimo 2.5

https://arena.ai/leaderboard/code/webdev/pareto

Good observation.

I actually started typing the same point that the chances are actually high because of train/eval overlap then realised you answered your own question with that same observation.

It is interesting though!

Perhaps in some way this means we should decide which eval set aligns best with our taste?

Back to the blog post. This is an excellent write up of an excellent technical achievement.

I have a lot of respect for the Cognition/Devin (always "Windsurf" to me) and Cursor teams.

I found it interesting - but justified - that they referred to themselves as a foundation lab rather than a dev tools company.

Agreed on the likely mechanism. I'm not sure "overfitting" is even the right description. These things are of course absurdly complicated, and evaluating their quality down to a single number involves a lot of judgement and trade-offs. I think it's more "you get what you measure" which is true in human organizations too. Define a KPI and people work hard to make it go up, even if it's not quite right or has bad side-effects.

Composer 2.5 is worse than both; I use it all day for simple stuff. It's Kimi 2.6 in a new outfit.

Likely benchmaxxed. You see it in Qwen and other smaller models all the time.

> This tells me the benchmarks they use are cherry-picked.

Which benchmarks would you have chosen instead, and why?

And to do that you’ll need development so until we’re all out of a job they’ll keep pushing. Once automating is automated it’s done.

software and agentic workflows will obsolete those things

RL environments building on top of each other will get these models there

needs people doing software development lifecycles to figure it out and implement

Like diplomacy? heh

This isn't as easy as it sounds. Every ML model is struggling to balance between generalization and test performance.

Taking a good model like GLM5.2 and just fine tuning it on coding can decrease real world performance due to mechanics like catastrophic forgetting. There is also other interesting behaviors were training on a broad training set can improve coding performance because there is positive transfer.

There is 100% an effort to make solid coding focused models, but it is very hard to do that without including capabilities across a broad set of adjacent tasks.

Defining what "coding" means now, and how quickly we fall off the capability cliff seems increasingly important.

Today my "coding" sessions often enough begin with real life problems, where I discuss domain or inter-domain things, ranging from business, economics, psychology, etc. Being able to do all of that with one model is something I am willing to pay a premium for.

Of course not having to pay the premium, because the routing is smart or whatever, would be great. I just don't want to have to think about it.

Qwen was doing something like this with their coder models. But alas, they seem not to be releasing those anymore. Last one was Qwen3-coder-next.

And make it run locally!

The "Lightning" (Cerebras) variant isn't free, only the regular one, which runs closer to 50 TPS in my experience with SWE 1.6.

The 1000 TPS shows for me as "SWE 1.7 Lightning" and took 14% of my daily quota in one prompt on the $20 plan.

But the normal speed one seems to be free or with very generous limits.

I work with Devin daily (as well as Claude and a few others) and I can attest it's not a cheap product but it's a good one and it saves me a bunch of time.

This is inevitable when the primary incentive is to raise aggressively. Overall I dont find cognition blogs that jargony, there are definitely worse offenders

To be fair it does seem like most AI startups are now like this (particularly when it comes to constantly mentioning how hard they work and ignoring consumer markets).

Link for this?

I highly respect many people at cognition but yeah that's put a sour taste in my mouth.

I want to work in the AI space on actual AI research, at any part of the stack. Even if I'm developing training infra - as long as people are advancing knowledge of what intelligence could be.

But it seems like either it's big labs or grifters, that's it, and even the big labs, at least publicly, seem very grifty at times. Not like I have the technical chops probably, but still.

> "A company whose first demo was completely fraudulent"

Could you expand on this?

Would love to see these companies use benchmarks done by third parties.

What happened ?

Open weight models should have GPL-like license where it says if you train model on it, it needs to be open weight as well.

Yes, and not only that but you can't even access it via API, you can only use it in Devin (formerly Windsurf).

I'm an OpenCode user, but I'll fall back to Claude Code if I want to use Opus end to end for something, given my company has a subscription. But I'm not using yet another tool and subscription for a model that isn't even winning.

The regular one (not the fast variant) is free but slow. The "Lightning" variant (which uses Cerebras and gets supposedly 1000 TPS) costs $12.50/M output, $2.5/M input, $1/M cached input. So it's quite a bit more expensive than SWE 1.6.

and what is actual intelligence per dollar benchmark ???? its useless comparing token/dollar while some model inherently generate more thinking output and cost more despite lower cost

Why? I'm personally on the opposite end. Less babysitting/higher quality means more time goes back to me/the user. 1000tps of bad code means you have to keep validating the output and circling back.

Indeed. For me opus 4.8 is good enough. If only it would be 100 times faster. You could run it in self verification loops much much faster. It sometimes takes 15 minutes for me to complete a simple task. For example configuring AWS agentcore and deploying an agent on it. Takes forever with Claude with constant issues it tries to solve.

I really don't want harness lock-in. I am trying to decouple myself from Claude Code now. I love the model of OpenRouter and being able to switch models at will let's your harness focus on your personal tooling and you can easily switch to the flavor of the month LLM with a single slash command instead of rewiring your entire workflow to use a harness to use a model.

I like Cerabras, but I really wish they would make more of their hosted models generally available.

Harness-wrapper tools that support multiple harnesses and allow sharing workspace features (skills, slash commands, etc.) between them will be meta.

Ugh, that changes everything. If I wanted an arranged marriage I could go back to Claude Code.

Because papers are often referred to by the first author’s name, and often the first author is the primary researcher and therefore deserves the extra credit. When two or more primary authors are equally involved, they’ll often do a random ordering but annotate this so that no one thinks one did more than the others.

Pretty sure most benchmarks are being gamed by people training on the test set deliberately or accidentally anyway.

The reality is most people building their own models and providing that alongside SOTA ones don't really care about how great these models are. They just prove that 'hey we are smart enough to build our own models so you can trust us instead of going with a single provider like Claude via Claude Code', also a cheap alternative for cost sensitive/free users - at least this was the case for Windsurf, not sure if Devin Desktop still has that tier. They just need to hillclimb the benchmarks and show something reasonable enough there.

Benchmarks are just vibes with error bars... wake me up when it survives a week on a real codebase without hallucinating a package that doesn't exist.

Funny, the cheerleading at HN for leading Chinese models, but a non Chinese lab (building on top of a Chinese model) gets dissed here.

Is this open source? I can't find a link to download the weights.

Not open source. Also, not available beyond it's own harness.