Artificial Analysis is not trustworthy anymore. Every time an open-weight model, especially Chinese, looks to be nearing the top, AA updates the index to (ostensibly) specifically push closed models higher. Just recently with v5, OpenAI models were pushed up.

▲

asenna 3 hours ago | parent | next [-]

I don't think that's the right read. Deepseek Flash 4.1 1017 did not get affected much.

I think what's happening is Chines models genuinely try to benchmax on it and hence they suffer when a new version comes out.

	▲	disgruntledphd2 3 hours ago \| parent [-]
		> I think what's happening is Chines models genuinely try to benchmax on it and hence they suffer when a new version comes out. I read somewhere on Substack that a lot of this was driven by the inclusion of more knowledge work benchmarks, where the US labs are stronger, versus coding which is where most Chinese models appear to put their energies for now.

▲

cbg0 3 hours ago | parent | prev | next [-]

Or it could be that the benchmaxxed models are getting pushed down by the introduction of unsaturated benchmarks. Muse Wonder also went down 52 -> 43 after the update.

▲

jbellis 3 hours ago | parent | prev [-]

It's not that hard to create your own uncontaminated eval for the problems you care about, and compare.

Here's mine for coding in Rust, and I definitely see the same pattern of "most chinese models do much worse than on public evals that have been around long enough to train on" that AA v5 surfaces: https://slopcop.com/power-ranking