For anyone actually considering this $200k+ setup for home or even small business use the power usage and noise is extreme. 1.6kw each with the fans to match. 600tokens/s at concurrency 16 is not what you need for home use or even sub 100 employee business use either. Really reconsider please (do you seriously need people to tell you this?). I get tech wages are high and you can probably afford this but seriously.

Right now there’s no great solution to the frontier or near frontier models for home use. But there are cards and options coming in 2027. For now just accept that you run qwen 37b or similar.

What’s coming in 2027 that changes this and why i implore you to wait? Well for these models you need massive amounts of memory and memory bandwidth. The solution above gets this by using expensive HBM spread over enough cards with enough compute to give a ridiculous tok/s. For home use you’d actually be happier with a single GPU that had lots of cheap memory to hold the weights and just enough memory bandwidth to feed that single gpu and no more. This in theory gets you close to 50tok/s on GLM 5.3 if it existed.

In 2027 there’s expected to be multiple HBF (high bandwidth flash) options incoming. High bandwidth flash has ~10x the latency of HBM but that actually doesn’t matter much for inference. Inference is latency tolerant and local inference time to first token beats over the internet time to first token no matter what you do in any case. The bandwidth and amount of memory is the blocker.

In 2027 there’s expected to be many multi terabyte HBF options coming. NAND/NOR flash is massively cheaper than RAM per byte (multiple states per logical unit so it’s denser) and it’s not that hard to wire it in a more parallel way to increase its bandwidth. Nvidia has the Kioxia solutions and Sandisk, Samsung, Tenstorrent, Mythic have also previewed HBF with GPUs to do local inference on the open source frontier models at reasonably high tok/s. Even if HBF is sold for 10x the price per terabyte of todays flash that’s still an extremely small fraction of the price of any other option right now.

Make do with the subscriptions for now. Don’t go crazy trying to run the frontier at home today. We should have something much better solutions are coming.

Make do with the subscriptions for now.

Sometimes you can't. All big three are bringing in the censorship and virtue signaling. It got to absurd levels recently. I asked claude to audit my code, let's see specific security features and it flat out refused etc.

I did a janky setup at home - I hooked up an old razer egpu case with 3090 (which had been collecting dust) to a strix halo 395 machine. I'm running qwen 3.8 flash next abliterated version as well as few image editing models and hook up to it with pi. The speed is really decent, it works on a level like frontier models late last year or better, and the best thing is I feel I can do anything with it without being nannied around.

I still use frontier, but now I use both. I recommend everyone to get one of these machines and do a similar setup. I got the idea from here anyways.