| |
| ▲ | simonw 2 days ago | parent | next [-] | | I've been calling that the "streaming experts" trick, the key idea is to take advantage of Mixture of Expert models where only a subset of the weights are used for each round of calculations, then load those weights from SSD into RAM for each round. As I understand it if DeepSeek v4 Pro is a 1.6T, 49B active that means you'd need just 49B in memory, so ~100GB at 16 bit or ~50GB at 8bit quantized. v4 Flash is 284B, 13B active so might even fit in <32GB. | | |
| ▲ | zozbot234 2 days ago | parent | next [-] | | The "active" count is not very meaningful except as a broad measure of sparsity, since the experts in MoE models are chosen per layer. Once you're streaming experts from disk, there's nothing that inherently requires having 49B parameters in memory at once. Of course, the less caching memory does, the higher the performance overhead of fetching from disk. | |
| ▲ | EnPissant 2 days ago | parent | prev | next [-] | | Streaming weights from RAM to GPU for prefill makes sense due to batching and pcie5 x16 is fast enough to make it worthwhile. Streaming weights from RAM to GPU for decode makes no sense at all because batching requires multiple parallel streams. Streaming weights from SSD _never_ makes sense because the delta between SSD and RAM is too large. There is no situation where you would not be able to fit a model in RAM and also have useful speeds from SSD. | | |
| ▲ | simonw 2 days ago | parent | next [-] | | There have been some very interesting experiments with streaming from SSD recently: https://simonwillison.net/2026/Mar/18/llm-in-a-flash/ | | |
| ▲ | EnPissant 2 days ago | parent [-] | | I don't mean to be a jerk, but 2-bit quant, reducing experts from 10 to 4, who knows if the test is running long enough for the SSD to thermal throttle, and still only getting 5.5 tokens/s does not sound useful to me. | | |
| ▲ | simonw 2 days ago | parent [-] | | It's a lot more useful than being entirely unable to try out the model. | | |
| ▲ | EnPissant 2 days ago | parent [-] | | But you aren't trying out the model. You quantized beyond what people generally say is acceptable, and reduced the number of experts, which these models are not designed for. Even worse, the github repo advertises: > Pure C/Metal inference engine that runs Qwen3.5-397B-A17B (a 397 billion parameter Mixture-of-Experts model) on a MacBook Pro with 48GB RAM at 4.4+ tokens/second with production-quality output including tool calling. Hiding the fact that active params is _not_ 17B. | | |
| ▲ | simonw a day ago | parent [-] | | It doesn't have to be a 2-bit quant - see the update at the bottom of my post: > Update: Dan's latest version upgrades to 4-bit quantization of the experts (209GB on disk, 4.36 tokens/second) after finding that the 2-bit version broke tool calling while 4-bit handles that well. That was also just the first version of this pattern that I encountered, it's since seen a bunch of additional activity from other developers in other projects. I linked to some of those in this follow-up: https://simonwillison.net/2026/Mar/24/streaming-experts/ |
|
|
|
| |
| ▲ | inventor7777 2 days ago | parent | prev | next [-] | | On Apple Silicon Macs, the RAM is shared. So while maybe not up to raw GPU VRAM speeds, it still manages over 450GB/s real world on M4 Pro/Max series, to any place that it is needed. They all do have a limitation from the SSD, but the Apple SSDs can do over 17GB/s (on high end models, the more normal ones are around 8GB/s) | | |
| ▲ | EnPissant 2 days ago | parent [-] | | Yeah, I am mostly only talking about the SSD bottleneck being too slow. No way Apple gets 17GB/s sustained. SSDs thermally throttle really fast, and you have some random access involved when it needs the next expert. |
| |
| ▲ | 2 days ago | parent | prev [-] | | [deleted] |
| |
| ▲ | zargon 2 days ago | parent | prev | next [-] | | > ~100GB at 16 bit or ~50GB at 8bit quantized. V4 is natively mixed FP4 and FP8, so significantly less than that. 50 GB max unquantized. | |
| ▲ | inventor7777 2 days ago | parent | prev [-] | | Ahh, that actually makes more sense now. (As you can tell, I just skimmed through the READMEs and starred "for later".) My Mac can fit almost 70B (Q3_K_M) in memory at once, so I really need to try this out soon at maybe Q5-ish. |
| |
| ▲ | zozbot234 2 days ago | parent | prev | next [-] | | These are more like experiments than a polished release as of yet. And the reduction in throughput is high compared to having the weights in RAM at all times, since you're bottlenecked by the SSD which even at its fastest is much slower than RAM. | |
| ▲ | the_sleaze_ 2 days ago | parent | prev [-] | | Do you have the links for those? Very interested | | |
|
