|
| ▲ | Aurornis 3 hours ago | parent | next [-] |
| Running larger-than-RAM LLMs is an interesting trick, but it's not practical. The output would be extremely slow and your computer would be burning a lot of power to get there. The heavy quantizations and other tricks (like reducing the number of active experts) used in these demos severely degrade the quality. With 64GB of RAM you should look into Qwen3.5-27B or Qwen3.5-35B-A3B. I suggest Q5 quantization at most from my experience. Q4 works on short responses but gets weird in longer conversations. |
| |
| ▲ | kgeist an hour ago | parent | next [-] | | >I suggest Q5 quantization at most from my experience. Q4 works on short responses but gets weird in longer conversations. There are dynamic quants such as Unsloth which quantize only certain layers to Q4. Some layers are more sensitive to quantization than others. Smaller models are more sensitive to quantization than the larger ones. There are also different quantization algorithms, with different levels of degradation. So I think it's somewhat wrong to put "Q4" under one umbrella. It all depends. | | |
| ▲ | Aurornis 37 minutes ago | parent [-] | | I should clarify that I'm referring generically to the types of quantizations used in local LLM inference, including those from Unsloth. Nobody actually quantizes every layer to Q4 in a Q4 quant. |
| |
| ▲ | freedomben 2 hours ago | parent | prev [-] | | I've tried a number of experiments, and agree completely. If it doesn't fit in RAM, it's so slow as to be impractical and almost useless. If you're running things overnight, then maybe, but expect to wait a very long time for any answers. | | |
| ▲ | zozbot234 2 hours ago | parent [-] | | Current local-AI frameworks do a bad job of supporting the doesn't-fit-in-RAM case, though. Especially when running combined CPU+GPU inference. If you aren't very careful about how you run these experiments, the framework loads all weights from disk into RAM only for the OS to swap them all out (instead of mmap-ing the weights in from an existing file, or doing something morally equivalent as with the original MacBook Pro experiment) which is quite wasteful! This approach also makes less sense for discrete GPUs where VRAM is quite fast but scarce, and the GPU's PCIe link is a key bottleneck. I suppose it starts to make sense again once you're running the expert layers with CPU+RAM. |
|
|
|
| ▲ | anemll 2 hours ago | parent | prev | next [-] |
| Yes, SSD speed is critical though. The repo has macOS builds for CLI and Desktop.
It's early stages though. M4 Max gets 10-15 TPS on 400B depending on quantization. Compute is an issue too; a lot of code is PoC level. |
|
| ▲ | jnovek 3 hours ago | parent | prev [-] |
| I have a 64G/1T Studio with an M1 Ultra. You can probably run this model to say you’ve done it but it wouldn’t be very practical. Also I wouldn’t trust 3-bit quantization for anything real. I run a 5-bit qwen3.5-35b-A3B MoE model on my studio for coding tasks and even the 4-bit quant was more flaky (hallucinations, and sometimes it would think about running tools calls and just not run them, lol). If you decided to give it a go make sure to use the MLX over the GGUF version! You’ll get a bit more speed out of it. |
