Remix.run Logo
▲ maplet 2 hours ago

It scores a 43 on Artificial Analysis with an average cost of $3.27 [1]. For reference, GPT-6.1 Sol (medium) gets a 46 with an average of $0.36.

[1]: https://artificialanalysis.ai/models/qwen4-max

▲esperent 2 hours ago | parent | next [-]

The table on that page shows gpt 6.1 sol (max) at $1.27 per task and it's the only model cheaper than Qwen 4 max.

By comparison opus 5.5 (max) is $10.01.

If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.

▲Aurornis 34 minutes ago | parent | next [-]

The Artificial Analysis charts show max effort by default, which isn't how most people use them.

Scroll down to the cost per task pareto front chart which includes more difficult levels by default: https://artificialanalysis.ai/models/qwen4-max#intelligence-...

Qwen4-Max is not in a good spot on that chart. Astra xhigh and Fable 5.2 medium had a similar cost per task with higher intelligence ratings.

▲esperent 25 minutes ago | parent | next [-]

I cannot see cost per task for any other thinking levels for Qwen 4 Max.

If it's actually there, what is the cost at medium thinking? It would show Qwen 4 Max (medium) if it's there.

Everywhere that I can see on the page it just shows ”Qwen 4 Max" while for 6.1 Sol, Opus 5.5 there's always a thinking level in brackets after: (low), (medium), (max) etc.

▲SV_BubbleTime 32 minutes ago | parent | prev [-]

How “safe” is Qwen4? Will it immediately shut down on security research like the US frontier models?

▲lcnPylGDnU4H9OF an hour ago | parent | prev [-]

> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.

They compare different reasoning levels.

https://artificialanalysis.ai/models/deepseek-v4-1-flash-101...

▲seizethecheese 2 hours ago | parent | prev | next [-]

Does anyone really believe the cost data for benchmarks? Pretty sure all models will behave differently at this point when they get benchmark shaped questions.

(And I’m sitting on data showing Sol 6.1 and Qwen 3.8 Max cost roughly the same per task, for real tasks.)

▲eli an hour ago | parent | next [-]

The cost per task metric is accurate for those benchmark tasks. If your tasks don’t look like benchmark tasks..you should use your own benchmarks :)

▲XCSme 39 minutes ago | parent [-]

Lol, I did that with https://aibenchy.com

▲killingtime74 an hour ago | parent | prev [-]

I mean not believing intelligence score is reasonable but cost is just reporting what they spent?

▲seizethecheese an hour ago | parent [-]

I believe low cost but not high cost, since a model will work really hard when a question looks benchmark-like.

▲scotty79 2 minutes ago | parent | prev | next [-]

Looks like a slightly weaker, cheaper Kimi.

▲thefourthchime 12 minutes ago | parent | prev | next [-]

[dead]

▲augment_me 14 minutes ago | parent | prev | next [-]

Artificial Analysis is a VC-backed firm and it's unclear what the incentives are there.

They have altered their benchmark multiple times in the last 4 weeks (v5 in https://artificialanalysis.ai/articles/artificial-analysis-i...) and the last 5.0.1 change https://artificialanalysis.ai/methodology/intelligence-bench... pushed up Anthropic's models a bit, you can see that v5.0.0 set on the first link Opus 5.5 has a lower score than in v5.0.1, and v5.0.1 was released just before Fable dropped.

Some part of me believes that the benchmark mix was put together to favor Anthropic's models.

▲montroser 2 hours ago | parent | prev | next [-]

Yeah, it's in the same ballpark as Kimi 3.1. Both of which are obviously open weight, so that's at least a big differentiator from Sol. Let's hope against hope that the much smaller flash model might beat deepseek v4.1 flash and run for as cheap.

▲jrflo an hour ago | parent | prev [-]

It looks pretty bad on those benchmarks. Is it possible that this is due to the recent crackdown on distillation?

▲raincole 13 minutes ago | parent [-]

I think it's the most plausible explanation.