Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

This is great. I've been building my own model benchmark lately and it has indeed been so easy to mess up the scoring. It's simply much harder to come up with an algorithm that combines all your individual scores into something that isn't broken in some special circumstances. That's why I think many just start capping the results.

▲

stephantul 4 hours ago | parent | prev | next [-]

The (relevant) segue into tires elevates this post so much. I don’t understand why, but it does

▲

IsTom 2 hours ago | parent | next [-]

I'm a little bit confused about these tire claims as

> down to 0C / 32 F (he didn't test colder conditions)

I get that people live in different places, but that's a huge caveat. How's that winter if you're not below 0°C? That sounds like "winter tires are worse than all-seasons tires in winter if you exclude winter".

	▲	pm215 10 minutes ago \| parent [-]
		I agree that testing colder conditions would be useful, but I guess you get into problems with it being ice and snow that you're testing, not the "summer tyres get too hard in the cold" hypothesis. It's winter if it's noticeably cooler than summer. For instance London's climate has a January average minimum temperature of just under 3C, which is very clearly winter compared to the July minimum of 14C. (figures from Met office site on their Heathrow station.) While the temperature does drop below 0C sometimes, it is not consistently below zero. In southern England pretty much nobody changes tyres for winter -- you just use the same set all year. Optimising for "2C in the wet" seems about right...

▲

vintagedave 3 hours ago | parent | prev [-]

Maybe that it's an example of the same pattern in a real-world, non-digital area? It makes it feel more grounded.

▲

jdw64 an hour ago | parent | prev [-]

I feel like AI development is hitting a wall now. Evaluations are driven by benchmarks, but I have no idea who is even evaluating the validity of these benchmarks. The idea that continuously training and scaling models will automatically yield broad capabilities across multiple domains hardly seems true anymore.

Benchmarks like Senior SWE-Bench apply discontinuous and arbitrary thresholds to evaluate results. If the generated code is semantically identical to the reference solution but even slightly exceeds a length cutoff, it fails? That seems like a genuinely flawed criterion, and the LLM-based graders themselves feel highly unstable.

Honestly, my impression is that LLM advancement is now completely dictated by benchmarks. But looking at recent trends, token prices are skyrocketing while coding capabilities haven't shown any massive improvements beyond a certain model generation. In fact, comparing GPT-6 Astra to 5.6 SOL, SOL often writes better code.

Considering all this, while training across diverse domains can pack a model with various pieces of knowledge, ultimately, it feels impossible for this approach to do something like derive the theory of relativity from medieval knowledge.

	▲	lh712 31 minutes ago \| parent [-]
		Nitpicking: It is not possible to derive the theory of relativity from medieval knowledge. At least not in any way resembling how the development actually happened, that is, heavily influenced by observations (or rather an iteration of speculation and observation). [Unless one counts both Galilean relativity and some parts of the theory of electromagnetism as being contained in medieval knowledge.] Perhaps it would be possible if the classically observed reality turned out to be possible (consistent under some reasonable conditions) only as a consequences of some deeper, sufficiently determined theory; but that would go far beyond our current level of knowledge, at least as far as I can say.