How do you use a 72GB model as your daily driver locally?

There was a 3.5 122B 10A release -

https://huggingface.co/Qwen/Qwen3.5-122B-A10B

I've heard 27B is smarter! I tried it some time ago but couldn't get it working with my oMLX. I need to try it again.

On an 8GB GPU and 32GB laptop: ~5 words/s while running in Qubes via ollama (I don't have an install at the moment that'll tell me tokens/s). Not exactly a highly tuned setup, but it's a ballpark at least :)

Tolerable and usable for some things, though thinking makes it take about a minute to reply in many cases. But getting this kind of thing to run on 8GB of VRAM is the main benefit of the mix-of-experts setup: it can do partial GPU loading and get a ton better throughput than a similarly-sized dense model.

I use it with a strix halo server. 35B runs stupidly fast. 27B is about 700 TPS prefill and 30 TPS token generation. Which interestedly is about what Kimi K3 gives me depending on provider.

On my SpacemiT K3 SBC with 32GB RAM (where models run on the eight A100 RISC-V cores with 1024 bit vectors) doing the same task I got 5, 5.8, 6.5 tok/s using gemma-4-26B-A4B-it-QAT-Q4_0.gguf, Qwen3.6-35B-A3B-Q4_K_M.gguf, Qwen3.5-35B-A3B-Q4_K_M.gguf. The corresponding dense models are more in the 2.5-3 tok/s range.

Kind of slow, but using only 14W of electricity so the Wh per task is twice as good as using my i9-13900 laptop with 4060 GPU.

It's pretty fast, faster than I could type anyway, but not as fast as Claude of course. My oMLX dashboard says I get about 45 tokens per second from the Qwen model I'm running (I host it on my M1 Mac Studio, not on my laptop).

It's just a MacBook Air with an M4, cheap and nothing special. I host Qwen on my Mac Studio, an M1 with 64gb ram. The model uses around 20-25gb ram depending on what it's doing.