Personally I find speculative decoding much better strategy than MoE – performance wise it's there at 90-100 t/s on 2x4090, great intelligence – really great fit.

▲

d4rkp4ttern 3 hours ago | parent | next [-]

A lot of people, including me, don’t want to bother with GPUs, they’d rather run it on their M1-M5 MacBook. For example the 35B-A3B is very usable even on a M1 64GB MacBook.

▲

mirekrusin 2 hours ago | parent [-]

Speculative decoding also works on Mac, 64G is more than what I have, m5 max should handle up to ~40 t/s with optimized setup (and with a lot of vram you can get great wins on concurrency – that harness can take advantage of for single user task as well), but agree memory bandwidth in mac or spark is still too slow, next gen for both will be great hardware to have for sure.

	▲	smcleod an hour ago \| parent [-]
		I get around 70tk/s on the m5 max, with 5bit AWQ / oQ5 slowing only to around 40tk/s at higher context.

▲

c0m47053 2 hours ago | parent | prev [-]

MoE is great on systems that lack the VRAM to host the full model. On my 16GB VRAM system, I can get 100 tok/s with Q4 Qwen 3.6 35b a3b, and 15 tok/s with 27b.

MTP is a trade-off, as it pushes some more of the model off the GPU.

I have managed to get usable quants of Laguna S2 and even DeepSeek V4 flash on this setup.

There is clearly some intelligence loss compared to similar sized dense models, but I feel like it stomps on the 9-12b models I could run fully on GPU