On artificialanalysis.ai, Kimi 2.7 Code is way worse than GLM 5.2 at everything (general intelligence, coding, agentic tasks).

But here, both Kimi 2.7 and its derivative SWE-1.7 are ahead of GLM 5.2. This tells me the benchmarks they use are cherry-picked.

▲

qingcharles an hour ago | parent | next [-]

Composer 2.5 is worse than both; I use it all day for simple stuff. It's Kimi 2.6 in a new outfit.

▲

bearjaws 3 hours ago | parent | prev | next [-]

Likely benchmaxxed. You see it in Qwen and other smaller models all the time.

▲

petesergeant 4 hours ago | parent | prev [-]

> This tells me the benchmarks they use are cherry-picked.

Which benchmarks would you have chosen instead, and why?

▲

cogman10 2 hours ago | parent [-]

It honestly seems like there's not a great way to currently benchmark AI.

The ideal way to run these benchmarks would be to give a 3rd party the model to run in an isolated environment so the prompts don't make their way back to the AI engineers.

That seems doable for open weight models, but not for private models.

	▲	p1necone 35 minutes ago \| parent [-]
		If you've got money to burn on tokens, the way that seems best to me is to set up a repeatable harness - docker container with a specific past commit from your own project, set of known issues/features that you've already fixed/completed of varying levels of complexity. Set up a script that launches the harness for each model, prompts them to implement one of the tasks, let it churn until either tests pass or it hits some budget limit. Then, most importantly, read the transcript and output and judge subjectively - I don't think this actually can be narrowed down to a score, although tokens burned to fix, whether it actually got the tests green etc are all good signals. (I've done this, but so far only on a codebase that was too complicated with models that were too weak because I didn't want to spend more than a few dollars - results were inconclusive, planning on iterating on my personal benchmark in future)