| ▲ | seanmcdirmid 2 hours ago | |||||||||||||
That MoE model is fast. 30B A3B gets 140 tokens/second on my M3 Max, and…the A3B part means a reasonable KV cache size and reasonable prefill times. 3.8 27B is a dense model that is mostly unusable, 40 tokens/second using MTP sure, but any cache miss is a few minutes of prefill. You are better off going with a lobotomized 3.8 flash MoE for the same amount of memory. The report is still bizarre, but that model is still very relevant to local users. | ||||||||||||||
| ▲ | rockdoe an hour ago | parent [-] | |||||||||||||
I have no idea why you wouldn't be using Qwen3.6 35B-A3B with that setup. Or did you mistakenly think that was what they're talking about? Their model is over a year old, like, a few decades in LLM history. | ||||||||||||||
| ||||||||||||||