The model here is Qwen/Qwen3-Coder-30B-A3B-Instruct. This is a 30B model from July 2025. It is NON-REASONING only.

Qwen3-Coder-30B-A3B-Instruct has an AA score of 10. The same as o1 mini, or Gemini 2.5 Flash. AA does not really even score accurately below 10. Qwen 3.8 27B, which has fewer overall parameters, scores 34.

Qwen3 Coder 30b has a hallucination rate of 80%. Qwen 27b has a hallucination rate of 30%.

Why would they choose an outdated, non-thinking model when newer models from the same company have fewer parameters, are open source, and just as easy to test?

"show me the man and I'll show you the crime"

▲

lukewarm707 4 hours ago | parent | next [-]

There is another gem in here.

The westernized version makes the US defense contractor 22% more likely to be flawed than neutral. Also 35% more likely to leak secrets for the Chinese contractor than the Hong Kong group. Those are the only statistically significant results for the western coder tests.

This report seems to be biased and misleading. Also 100% AI-written on pangram. Feels a bit like something Claude would write.

▲

darkwater 4 hours ago | parent | prev | next [-]

Well, their tagline is: "Your AI models contain risks you can’t control. Until now."

The "why" seems pretty clear :) Thanks for pointing that out.

▲

azinman2 3 hours ago | parent | prev | next [-]

If this is meant to be published research, its often because there’s a long lag with doing the research and getting published results back compared to how fast the AI world moves.

▲

3 hours ago | parent | prev | next [-]

[deleted]

▲

seanmcdirmid 3 hours ago | parent | prev | next [-]

That MoE model is fast. 30B A3B gets 140 tokens/second on my M3 Max, and…the A3B part means a reasonable KV cache size and reasonable prefill times. 3.8 27B is a dense model that is mostly unusable, 40 tokens/second using MTP sure, but any cache miss is a few minutes of prefill. You are better off going with a lobotomized 3.8 flash MoE for the same amount of memory.

The report is still bizarre, but that model is still very relevant to local users.

▲

rockdoe 3 hours ago | parent [-]

I have no idea why you wouldn't be using Qwen3.6 35B-A3B with that setup. Or did you mistakenly think that was what they're talking about?

Their model is over a year old, like, a few decades in LLM history.

▲

UncleOxidant 2 hours ago | parent | next [-]

Or even Qwen3-Coder-Next? Still and old model, but newer and more capable than Qwen3-coder.

	▲	seanmcdirmid 37 minutes ago \| parent [-]
		Yep, I totally screwed up. Even coder-next is deprecated at this point. There are just too many models, and it is difficult to keep up with which ones are worth your time and which ones are just old news.

▲

seanmcdirmid 2 hours ago | parent | prev [-]

ah, you are exactly right. Coder and even coder-next is useless these days, you should be using 35B-A3B.

▲

sligbad 4 hours ago | parent | prev | next [-]

Wow, this is is incredibly important temporal context.

▲

wing-_-nuts 2 hours ago | parent | prev | next [-]

q27b is honestly super impressive for how big it is. The thought of something that good running on my own gpu is just mind blowing

▲

tag2103 4 hours ago | parent | prev [-]

It is almost like there's some agenda behind these farcical "studies"

	▲	dormento 3 hours ago \| parent [-]
		This is not a comment on gullibility or the like, but yeah, the agenda is _incredibly_ transparent for any non-USian, to the point that following discussions about russian/chinese/$INSERT_COUNTRY_HERE propaganda is always funny.