Remix.run Logo
▲ anana_ 2 hours ago

Artificial Analysis score doesn't look too hot. Doesn't even surpass Kimi 3.1

▲rbehrends 2 hours ago | parent | next [-]

While I doubt that Qwen 4 Max can seriously compete with frontier models, the AA index has become all but useless, due to the introduction of new benchmarks in the aggregate which are of questionable value as a measure of "intelligence" [1]. AA has seemingly also abolished updating the Coding Agents Index, which was an okay benchmark for coding ability, and older models are completely absent from the v5 version of the full index.

[1] Especially Terminal-Bench-Science 0.1 stands out. Example problems:

- https://hub.harborframework.com/tasks/terminal-bench-science...

- https://hub.harborframework.com/tasks/terminal-bench-science...

Probably very relevant to people working in these fields, but it is honestly not surprising if a lab with limited compute will not train for these technologies, and it says pretty much nothing about more common use cases. I also suspect that this benchmark measures knowledge retrieval more than raw reasoning ability. This is where you really need a breakdown of capabilities and not an increasingly meaningless aggregate number.

▲gardnr 3 minutes ago | parent | next [-]

Did AA take on funding or something?

▲izabera an hour ago | parent | prev [-]

alibaba has "limited compute"?

▲rbehrends an hour ago | parent [-]

Compared to OpenAI and Anthropic, yes. There's this export ban on Nvidia chips, after all. And all the CPUs they have won't help much with model training.

▲anon373839 2 hours ago | parent | prev | next [-]

AA has become pretty meaningless. They just keep adjusting the recipe to get a preferred ranking result, with no supporting evidence for their decisions.

Edit: In addition to reshuffling the weighting of individual benchmarks, they also do things like test Anthropic and OpenAI in their native harnesses, while not doing the same for Qwen/Deepseek/etc.

▲the_duke 2 hours ago | parent | next [-]

To be fair, a lot of the popular benchmarks have become pretty meaningless.

Models often get great scores despite being a lot worse than the frontier in practice.

▲anon373839 20 minutes ago | parent [-]

This is true, but if you look at the change in weighting to specific benchmarks from one version of AA to the next, it looks really arbitrary.

▲gr_norm 2 hours ago | parent | prev | next [-]

Yeah, I don't know why people bother with it anymore. It no longer reflects what you're likely to see in real-world usage, at least IME.

▲stratts 2 hours ago | parent | prev [-]

One thing I find confusing about AA is that they show GPT 6.1 dominating the lower end of the Pareto line, with eg, GPT 6.1 Sol at medium supposedly costing far less per task than DeepSeek V4.1 Flash at high.

Yet every time I compare the two in something like Pi, Sol ends up being 5-10x the price for a simple query, something like 7-10c vs. less than 0.5c for DeepSeek. What's the deal?

▲blackoil 2 hours ago | parent [-]

Same for me coding tasks taking $20-50 are easily done by deepseek in $5-10

▲selectodude 2 hours ago | parent | prev | next [-]

Anthropic finally figured out how to block them from distilling Opus.

▲m00dy 2 hours ago | parent [-]

how so ?

▲AAisthetiobeofA 2 hours ago | parent | prev [-]

> Artificial Analysis

Ah yes, the reverse-benchmaxers of OpenAI models.

▲yipinwong an hour ago | parent [-]

Ah yes, another new account without any backup to the claim.