| ▲ | oceanplexian 5 hours ago |
| If you want to do coding with a local LLM your best bet is a 6 year old Nvidia 3090 which is substantially more powerful than the highest end overhyped Apple product for 1/5th the price. |
|
| ▲ | chorizo 5 hours ago | parent | next [-] |
| That’s 24GB VRAM. Not enough to run a 27B model at a useful quant+context size. |
| |
| ▲ | nsbk 4 hours ago | parent | next [-] | | I beg to differ. Have a look at this repo with single/double 3090 optimized configs for Qwen and Gema models: https://github.com/noonghunna/club-3090 | |
| ▲ | 4 hours ago | parent | prev | next [-] | | [deleted] | |
| ▲ | sanderjd 4 hours ago | parent | prev | next [-] | | Yeah seems to me like the mac studios with the unified memory architecture are genuinely good bang for the buck at the moment, because of this memory size consideration? | |
| ▲ | SkitterKherpi 5 hours ago | parent | prev [-] | | You can run 8bit 27B models at 24GB, it's definitely enough for the model size. | | |
| ▲ | SwellJoe 4 hours ago | parent | next [-] | | The 8-bit quantized 27B Qwen 3.6 is 29GB. You absolutely cannot run that entirely on a 24GB GPU. You could run a 4-bit, which is 16-17GB. But, you'd need a smallish context or you'd need to quantize your KV cache. Something like TurboQuant or RotorQuant might help. 32GB is the lower bound for comfortably running this size model. I'd maybe even say 64GB is right-sized, because a 256k context is nice to have for agentic workflows, and that won't fit on a 32GB card without heavy quantization (but I haven't tried TurboQuant or RotorQuant to know what impact it has on memory use for context). You could also put some of the model into system RAM, but that defeats the purpose of your argument that a 3090 will outperform a Mac Mini or Mac Studio. If part of a dense model is in system RAM, it absolutely will not outperform a recent unified memory device. | | |
| ▲ | cpburns2009 3 hours ago | parent [-] | | A 32gb card does run it nicely. I use unsloth's UD-Q5_K_XL at 256k context (k/v at q8_0), and get ~67 t/s on a 5090. I still need to look into MTP. | | |
| |
| ▲ | barbacoa 3 hours ago | parent | prev | next [-] | | I'm running qwen 3.6 27b at 8bit quantization and 262k context. It takes 53gb of vram on my system. | |
| ▲ | bityard 4 hours ago | parent | prev | next [-] | | Quantization is a trade-off, though. The quality, while still perhaps good enough for many tasks, is not as good as the full 16-bit weights that the model was designed for/released with. | | | |
| ▲ | jnovek 4 hours ago | parent | prev [-] | | I think that’s only true for MoE models. A dense model like 3.6 27b will require more (plus a KV store). | | |
| ▲ | bityard 4 hours ago | parent [-] | | No, even MoE models need to fit into (V)RAM. MoE has faster inference because only a subset of layers are used to predict the next token, but the set of layers used changes with every token. |
|
|
|
|
| ▲ | iagooar 4 hours ago | parent | prev | next [-] |
| My problem is I won't accept anything lower than the 96GB the RTX Pro 6000 Blackwell has. My dream is a workstation with 2x Pro 6000 to run DeepSeek v4 Flash comfortably, possibly qwen 3.6 / ornith on turbo speed. But man, I have never purchased a computer which is more expensive than a decent family car. |
|
| ▲ | jnovek 5 hours ago | parent | prev | next [-] |
| An M1 Ultra has 800gbps unified memory. It’s nothing to do with Apple, it’s their microarchitecture. They’re just about the only game in town with high-bandwidth memory if you want >24GB (for less than $10k, anyway). |
| |
| ▲ | murderfs 3 hours ago | parent [-] | | A 5090 gets you 32GB with 1.8 TB/s of memory bandwidth for ~$4k, RTX A6000 gets you 48GB at 768 GB/s for ~$3.5k, 2x 3090 gets you 48GB for $2000 or so, and if you're willing to go into the wilderness, there are much cheaper options like the AMD MI50. |
|
|
| ▲ | dheera 3 hours ago | parent | prev [-] |
| 32GB V100 |
