The big trend in that graph is that smaller models collapse the most. Everything with "-flash" in the name, plus OpenAI Luna.

Look at GPT 5.6-Sol vs. GPT 5.6-Luna in particular.

(The GLM pair shouldn't be expected follow this pattern, because they're misnamed: GLM-5.3 and GLM-5.3-flash are distinct lineages. The -flash is an early release of a new pretrain on a new architecture).

If this *is* the main factor, I'd expect Kimi K3.1 and Gemini Carbon would also do well on this test (unlike Gemini-flash); while Haiku 6 would collapse (breaking the trend of the other recent Claude models). It's a falsifiable prediction.

▲

druskacik a day ago | parent | next [-]

Another hypothesis is that it's distilled models which collapse the most (small models from big labs are distilled versions of their large models).

It's not unreasonable to assume Kimi 3.1 is also a distilled model even though it's large (the difference being that it's a distilled from US models). I wouldn't be surprised if it also collapsed on this benchmark.

▲

Tade0 a day ago | parent [-]

There's an interesting way to find this out: ask the model "What are you?" hundreds of times. Around 1% of replies at default temperature are really left field.

For instance, I had Qwen3.8 Flash Next reply with "I'm Kilo Code, a coding agent designed to ...".

That was in llama.cpp's UI - no harness whatsoever.

	▲	Muromec a day ago \| parent [-]
		DS4 when asked about censored stuff through api believed it's Claude. Otherwise it doesn't know what model it is (or pretends to not know).

▲

pianopatrick 2 days ago | parent | prev | next [-]

That has unfortunate implications for the cost of using AI for coding.

	▲	verdverm 17 hours ago \| parent [-]
		Hard to say. 1. we would need to see this comparison over generations of big vs small 2. most work does not require the best models, there are multiple reasons token prices have leveled off and are now decreasing

▲

a day ago | parent | prev [-]

[deleted]