Remix.run Logo
▲ i2talics a day ago

Who has incentive to improve the quality of the benchmarks?

▲rat9988 a day ago | parent | next [-]

AI Labs actually. They would love the perfect benchmar representing real usecase. I'd even dare and guess that they don't share much publically because it is a huge competitive advantage.

▲adastra22 a day ago | parent [-]

> they would love the perfect benchmar

That right there is the problem.

▲stalfie 17 hours ago | parent | next [-]

It is "a" problem, but honestly not a huge one. Chasing benchmarks is an integral part of AI research, and it is how we have gotten this far, which is pretty damn far. Discovering the ways in which benchmarks fail is in itself an interesting process, and the amount of interesting new benchmarks has grown with the capabilities of the models as a result. The frontier labs might be a bit selective about which benchmarks they publish as marketing, but overall they have so far been reasonably diligent about avoiding outright cheating. Most of the truly benchmaxed models are not on the frontier (eg. see muse spark performance dropping off a cliff in the article).

This is a scenario where the interest of the users and the labs actually align perfectly, since labs that have people actually paying for their product want benchmarks to align as close as possible to real life performance. Otherwise they can't actually easily tell whether the new model they are developing is any good or not, and so risk wasting literal billions of dollars on training runs for a product that people won't end up paying for. A failed major training run is an absolute disaster at the frontier, GPT-4.5 for instance was an almost existential failure for OpenAI at the time.

Case in point, the most popular coding model, Opus 5.5, also just happens to be the only one that avoided the benchmaxing impulse in this article. This is the result of years of Anthropic working very hard to avoid it.

▲adastra22 7 hours ago | parent [-]

Your last paragraph undermines the first two.

▲rat9988 18 hours ago | parent | prev [-]

What is the problem?

▲RinTohsaka 16 hours ago | parent [-]

[flagged]

▲adastra22 a day ago | parent | prev | next [-]

The people who have no power over the situation: users.

▲s1artibartfast a day ago | parent [-]

Users always have the choice to ignore benchmarks or run their own

▲danuker a day ago | parent | next [-]

Exactly. Even a brief one (a single task done on two models) showed me GLM 5.3 Flash was superior to DeepSeek 4.1 Flash.

DS used nearly twice the budget to accomplish the same thing.

Ideally, these tasks should be something you actually need day to day.

▲lisplist an hour ago | parent [-]

I'd rank it DeepSeek 4.1 Flash 1017 > GLM 5.3 Flash > DeepSeek 4.1 Flash (Original) - 1017 is a huge improvement.

▲adastra22 20 hours ago | parent | prev [-]

That’s missing the point. Frontier models get shittier in certain ways with each release, due to benchmaxxing, and there is nothing we can do about that.

▲Bolwin a day ago | parent | prev [-]

The benchmark makers and anyone who relies on benchmarks

▲verdverm 14 hours ago | parent [-]

depends, are they optimizing for accuracy or something else?