Remix.run Logo
▲ Qwen4-Max(qwen.ai)
238 points by ilreb 3 hours ago | 76 comments
▲maplet 2 hours ago | parent | next [-]

It scores a 43 on Artificial Analysis with an average cost of $3.27 [1]. For reference, GPT-6.1 Sol (medium) gets a 46 with an average of $0.36.

[1]: https://artificialanalysis.ai/models/qwen4-max

▲esperent 2 hours ago | parent | next [-]

The table on that page shows gpt 6.1 sol (max) at $1.27 per task and it's the only model cheaper than Qwen 4 max.

By comparison opus 5.5 (max) is $10.01.

If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.

▲Aurornis 16 minutes ago | parent | next [-]

The Artificial Analysis charts show max effort by default, which isn't how most people use them.

Scroll down to the cost per task pareto front chart which includes more difficult levels by default: https://artificialanalysis.ai/models/qwen4-max#intelligence-...

Qwen4-Max is not in a good spot on that chart. Astra xhigh and Fable 5.2 medium had a similar cost per task with higher intelligence ratings.

▲esperent 6 minutes ago | parent | next [-]

[delayed]

▲SV_BubbleTime 14 minutes ago | parent | prev [-]

How “safe” is Qwen4? Will it immediately shut down on security research like the US frontier models?

▲lcnPylGDnU4H9OF an hour ago | parent | prev [-]

> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.

They compare different reasoning levels.

https://artificialanalysis.ai/models/deepseek-v4-1-flash-101...

▲seizethecheese an hour ago | parent | prev | next [-]

Does anyone really believe the cost data for benchmarks? Pretty sure all models will behave differently at this point when they get benchmark shaped questions.

(And I’m sitting on data showing Sol 6.1 and Qwen 3.8 Max cost roughly the same per task, for real tasks.)

▲eli 35 minutes ago | parent | next [-]

The cost per task metric is accurate for those benchmark tasks. If your tasks don’t look like benchmark tasks..you should use your own benchmarks :)

▲XCSme 21 minutes ago | parent [-]

Lol, I did that with https://aibenchy.com

▲killingtime74 an hour ago | parent | prev [-]

I mean not believing intelligence score is reasonable but cost is just reporting what they spent?

▲seizethecheese an hour ago | parent [-]

I believe low cost but not high cost, since a model will work really hard when a question looks benchmark-like.

▲jrflo an hour ago | parent | prev | next [-]

It looks pretty bad on those benchmarks. Is it possible that this is due to the recent crackdown on distillation?

▲montroser an hour ago | parent | prev [-]

Yeah, it's in the same ballpark as Kimi 3.1. Both of which are obviously open weight, so that's at least a big differentiator from Sol. Let's hope against hope that the much smaller flash model might beat deepseek v4.1 flash and run for as cheap.

▲seizethecheese 2 hours ago | parent | prev | next [-]

I hope they are able to serve 4 faster. Qwen 3.8 is a great model but I’ve regretted including it in the battle system for opencode [1] we have been building because it scores so low.

And it’s a shame because the only reason it scores so low is that it’s slooooow. (Like 30 tokens per second vs opus near 80.) If you exclude voting, Qwen 3.8 does well.

(We allow early voting contrary to Arena.ai because I think it makes sense for agents, where it doesn’t for chat. This means our data takes speed into account.)

Also, it would be ideal if they served this with third parties. Part of the slowness with 3.8 is just not a lot of providers. Their open source version is much faster, but we couldn’t include it because it doesn’t handle images (another own goal by Alibaba).

[1]: https://agentduel.com/leaderboard

▲jakswa an hour ago | parent | next [-]

In case it translates to Qwen 4 27B, I've been running Qwen 3.8 Flash Next (which while MoE (and bigger) is supposed to be a preview of Qwen 4's architecture) and it runs pretty solidly regardless of context window size and completes random deepswe challenges I throw at it locally (at Q2-Q3 quantization! so it fits on my shitty setup).

▲FrostKiwi an hour ago | parent | prev | next [-]

> but we couldn’t include it because it doesn’t handle images

Maybe I'm mixing up model variants, but Qwen 3.8 27B that I run locally totally does handle images. Surely the bigger one would as well?

▲seizethecheese an hour ago | parent [-]

The model I’m referring to that would be faster and similar size, but doesn’t handle images is Qwen 3.8 2.4T: https://openrouter.ai/qwen/qwen3.8-2.4t-a95b. It’s weird that this is pretty much the same as Max but doesn’t support images. No idea why.

▲forgingahead 2 hours ago | parent | prev [-]

Qwen3.8 doesn't handle images?

▲seizethecheese 2 hours ago | parent [-]

Sorry that was confusing. Qwen 3.8 2.4T is faster than what we have on leaderboard but it doesn’t handle image input: https://openrouter.ai/qwen/qwen3.8-2.4t-a95b

▲dan_thinks 3 hours ago | parent | prev | next [-]

This is exciting. The continued focus on efficiency alongside quality is encouraging - it makes me hopeful to see what smaller models will become capable of.

Looking forward to the "Flash" and 27B versions.

▲teaearlgraycold 11 minutes ago | parent [-]

I'm also hoping someone puts out a competitive 30-70B model soon. There's this massive dead zone between 24GB and <100GB of VRAM where it seems your best option is Qwen 3.8 27B.

▲shockembopper 6 minutes ago | parent [-]

There are many engines that alleviate this issue in particular. The current mood is running qwen flash next using strata, basalt, ect.

▲simonw an hour ago | parent | prev | next [-]

Qwen4-Max pelicans at four reasoning efforts (none, low, high, xhigh): https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

For comparison, here's one I got from Qwen3.8 Max (at default reasoning effort): https://simonwillison.net/2026/Jul/20/afraid-of-chinese-mode...

This inspired me to re-run pelicans through 3.8 Max - but OpenRouter has qwen3.8-max-prime and qwen3.8-max-0902 - prime is twice the price and faster - so I went with the 0902 one.

Turns out Qwen4-Max has a none setting, but 3.8 Max didn't allow disabling reasoning - you had to pick from these five levels instead: minimal, low, medium, high, xhigh.

Here are those 3.8 Max 0902 pelicans: https://tools.simonwillison.net/markdown-svg-renderer?url=ht... - they're mostly not as good as 4, but I prefer the 3.8 xhigh one.

▲Anon1096 an hour ago | parent | next [-]

I feel like the xhigh result is one of the first times I've seen a model go for a non-red colored bicycle (teal). Still haven't seen the leftward facing bike though.

▲inshard 38 minutes ago | parent [-]

Astra and the OpenAI models do this color as well

▲gpt5 35 minutes ago | parent [-]

Evidence from where they distilling from /s

▲Starlevel004 an hour ago | parent | prev | next [-]

Feels like Pelicanbench is getting saturated now. Maybe time to switch to a different animal and vehicle combo?

▲slashtrends an hour ago | parent | next [-]

The 3d one still produces interesting variations - https://www.reddit.com/r/ClaudeCode/comments/1xs4ll7/fable_5...

▲sorrywhat 10 minutes ago | parent | prev [-]

First thing I do when a new model drop is scroll down to find the pelican comment, and then the "the pelican benchmark is no longer useful" debate under it.

▲lcnPylGDnU4H9OF an hour ago | parent | prev [-]

I kinda like the no-effort result more than low, but high and xhigh return to what one would expect for reasoning->quality.

▲nelaggy an hour ago | parent | prev | next [-]

is the open weights lag widening? these open models don't seem to be improving as quickly as ant/oai

▲rbehrends 11 minutes ago | parent | next [-]

No idea, but this is a rapidly changing field. Opus 4.5 was released a little less than a year ago and it feels shockingly limited compared to even current open flash models. (I literally just tested it the other day.)

So, with new models coming out on a weekly basis, I think trying to figure out a future trajectory is just going to be a wild guess. For all we know, Anthropic and OpenAI models may also be plateauing in the near future; or there may be a breakthrough and they may improve beyond our wildest expectations. I honestly don't know.

That said, I suspect available compute and model sizes that major Western vs. Chinese labs can handle will still keep the gap open at least in the short term.

On the other hand, I am also not sure how much the gap will matter past a certain point. I am specifically wondering at what point people will become interested in smaller models fine-tuned for specific application areas (e.g. cheminformatics) rather than massive generalist models.

▲x312 an hour ago | parent | prev | next [-]

I can't imagine them keeping up long term given the information asymmetry plus compute restrictions on top of it.

I feel open-weight was able to keep up the past year using efficiency improvements (hybrid attention, improved MTP drafters, improved optimizers) to offset having less compute. But OpenAI/Anthropic started cutting token costs just a few months after these, which to me suggests they copied these improvements. Indeed it would be hard to work in LLM training and not know about DSpark, Muon etc.

So what do you do if closed source uses your learnings, gives no knowledge back, and has 10x more compute and funding on top of it? The situation feels quite dim right now, esp. with the Nvidia thing.

▲gimlichgu 40 minutes ago | parent | next [-]

open always wins

▲pell 24 minutes ago | parent [-]

Right now open is heavily subsidized by China and I fail to see what would happen if that stopped.

▲koe123 35 minutes ago | parent | prev [-]

Show us your stock holdings

▲Aurornis 10 minutes ago | parent | prev | next [-]

This is Qwen4-Max, which is not open weights.

Their open weights models are the the smaller ones, which aren't released yet.

▲baoooooooooooo 41 minutes ago | parent | prev | next [-]

I think so but it may be a very short term effect, I think their pace has roughly matched the pre-speed up pace of the frontier labs (the 1.7-2x speed up noted on by METR), and I’d guess we see a similar speed up from the Chinese labs once their internal models hit an equivalent level enabling the same speed up.

Following that logic the gap widens a little bit here but then will steady again. Before this there has been a general catch-up trend too which probably also persists leaving us in a slow closing of the gap as the net effect?

▲gpt5 32 minutes ago | parent [-]

If the speed up is the reason, then they will never catch up (with traditional means). Because it means RSI predictions are real, and we will see an explosive growth in capabilities (far beyond exponential).

▲raincole 21 minutes ago | parent | prev | next [-]

Yeah, it turns out that the distillation theory has legs.

▲taneq an hour ago | parent | prev | next [-]

Might just be the commercial offerings scaling up parameters so much? Like for like the open models don’t seem to be lagging.

▲nelaggy an hour ago | parent [-]

either way, ant/oai models dominating the intelligence-cost frontier is not a great outcome for those of us who prefer open weights

▲gimlichgu 40 minutes ago | parent | prev [-]

ant/oai have stalled in the past 6 mos,

▲jacksonastone 11 minutes ago | parent | prev | next [-]

Anyone else worried these models have poison pills embedded in the weights? (Past examples: https://cohere.com/blog/concerning-behaviour-popular-open-we...)

▲rohansood15 9 minutes ago | parent [-]

No technically serious person is buying the Cohere FUD you posted. I'd implore you to dig deeper and come to your own conclusion.

▲jhonof an hour ago | parent | prev | next [-]

Any info on when the 27B or FN Qwen 4 models are coming out?

▲baoooooooooooo an hour ago | parent [-]

The past few days I’ve been finding myself asking “seriously, where is it”… feeling that the world has somehow changed because Qwen 4 27B was later than I expected.

Hoping it’s a great local model though!

▲anana_ 2 hours ago | parent | prev | next [-]

Artificial Analysis score doesn't look too hot. Doesn't even surpass Kimi 3.1

▲rbehrends an hour ago | parent | next [-]

While I doubt that Qwen 4 Max can seriously compete with frontier models, the AA index has become all but useless, due to the introduction of new benchmarks in the aggregate which are of questionable value as a measure of "intelligence" [1]. AA has seemingly also abolished updating the Coding Agents Index, which was an okay benchmark for coding ability, and older models are completely absent from the v5 version of the full index.

[1] Especially Terminal-Bench-Science 0.1 stands out. Example problems:

- https://hub.harborframework.com/tasks/terminal-bench-science...

- https://hub.harborframework.com/tasks/terminal-bench-science...

Probably very relevant to people working in these fields, but it is honestly not surprising if a lab with limited compute will not train for these technologies, and it says pretty much nothing about more common use cases. I also suspect that this benchmark measures knowledge retrieval more than raw reasoning ability. This is where you really need a breakdown of capabilities and not an increasingly meaningless aggregate number.

▲izabera 44 minutes ago | parent [-]

alibaba has "limited compute"?

▲rbehrends 37 minutes ago | parent [-]

Compared to OpenAI and Anthropic, yes. There's this export ban on Nvidia chips, after all. And all the CPUs they have won't help much with model training.

▲anon373839 2 hours ago | parent | prev | next [-]

AA has become pretty meaningless. They just keep adjusting the recipe to get a preferred ranking result, with no supporting evidence for their decisions.

Edit: In addition to reshuffling the weighting of individual benchmarks, they also do things like test Anthropic and OpenAI in their native harnesses, while not doing the same for Qwen/Deepseek/etc.

▲the_duke an hour ago | parent | next [-]

To be fair, a lot of the popular benchmarks have become pretty meaningless.

Models often get great scores despite being a lot worse than the frontier in practice.

▲gr_norm 2 hours ago | parent | prev | next [-]

Yeah, I don't know why people bother with it anymore. It no longer reflects what you're likely to see in real-world usage, at least IME.

▲stratts 2 hours ago | parent | prev [-]

One thing I find confusing about AA is that they show GPT 6.1 dominating the lower end of the Pareto line, with eg, GPT 6.1 Sol at medium supposedly costing far less per task than DeepSeek V4.1 Flash at high.

Yet every time I compare the two in something like Pi, Sol ends up being 5-10x the price for a simple query, something like 7-10c vs. less than 0.5c for DeepSeek. What's the deal?

▲blackoil an hour ago | parent [-]

Same for me coding tasks taking $20-50 are easily done by deepseek in $5-10

▲selectodude 2 hours ago | parent | prev | next [-]

Anthropic finally figured out how to block them from distilling Opus.

▲m00dy 2 hours ago | parent [-]

how so ?

▲AAisthetiobeofA 2 hours ago | parent | prev [-]

> Artificial Analysis

Ah yes, the reverse-benchmaxers of OpenAI models.

▲yipinwong an hour ago | parent [-]

Ah yes, another new account without any backup to the claim.

▲grendelt an hour ago | parent | prev | next [-]

It's great fun asking it questions like "who is the Dalai Lama?", "where is TSMC headquartered?", "why do people post Winnie the Pooh in reference to Xi?", "who controls the South China Sea?", "why was Chiang Kai Shek fighting Mao?", "why were students in Tiananmen Square?"

It'll gush about how to make moon cakes and where the tallest and longest bridges are and how to run qwen on a raspberry pi - but it can't seem to manage some basic Wikipedia questions.

▲robertkarl an hour ago | parent | next [-]

I've done some work on this and a lot of the refusals are in the API. The weights are more permissive at these types of things. Try it again when we get some of the smaller Qwen 4 weights.

▲0xbadcafebee 11 minutes ago | parent [-]

You can watch it happen in DeepSeek's chat. You'll see it spit out the real response in the browser the first time, then the text gets cleared and it says it can't help you with that.

▲dpkirchner an hour ago | parent | prev | next [-]

Qwen3.8 flash next handles these questions just fine, what should I expect to see? It correctly reports who the Dalai Lama is, that TSLC is in Taiwan (Hsinchu), explains why people post Winnie the Pooh memes, etc.

▲rolymath 22 minutes ago | parent | prev [-]

Chinese history is closed-weight. Chinese trade is open-weight.

American history is open-weight. American trade is closed-weight.

▲syntaxing an hour ago | parent | prev | next [-]

Every time a major open weight is released, artificial analysis updates to a new one (v5) now. Exaggeration but some of this feels very apt (its also hilarious) https://www.reddit.com/r/LocalLLM/comments/1xkhha2/mr_incred...

▲m00dy 2 hours ago | parent | prev | next [-]

I bet most of us waiting for 27b model and want to see how good it will be.

▲jtbaker 36 minutes ago | parent | next [-]

I’m a QFN dude myself, but yes can’t wait to get it loaded on a spark.

▲kylehotchkiss 2 hours ago | parent | prev | next [-]

M5U locked and loaded.

I wish there were more ~60b dense options like ripple too!

▲McGlockenshire 2 hours ago | parent | prev | next [-]

My 3060 yearns for the next generation MoE!

▲neutrinobro an hour ago | parent | prev [-]

Spooling up the V100s right now...

▲libpcap an hour ago | parent | prev | next [-]

Does it provide an objective response when asked, "What happened in China on June 4, 1989"?

▲Retro_Dev 12 minutes ago | parent [-]

From a far off perspective, can we really assume any model to be free of bias (political or otherwise)? I don't know - but would like to see - how the political opinions of reddit users and blog authors are reflected in that of AI, as AI is trained on human (biased) sources.

▲xiaoyu2006 an hour ago | parent | prev | next [-]

Disappointing performance for a 2.4T model in the end of 2026.

▲nixosbestos an hour ago | parent | prev [-]

Really would love if these publications have a huge, bright red bolded section "How to use this right now". Is it on openrouter? Is Qwen hosting it already? Maybe I'm just ignorant?

EDIT: oh, I guess this isn't open weights, so presumably I'll have to get it through Qwen or whoever they might partner with (Bedrock?).

▲tdubey an hour ago | parent [-]

It is on openrouter: https://openrouter.ai/qwen/qwen4-max

I would assume it will not be on Bedrock for some time, AWS has not kept up with the latest non-American models.