32GB of fast unified memory is enough for Qwen 3.8 27B.

- 16GB for the weights at Q4

- 9GB for the full 256K context at Q8

- 7GB spare for overhead and system.

The problem is that these Macs have 32GB of slow unified memory.

Edit: I'm thinking of a headless Mac mini, if you meant running it on the same machine you're using of course you'll need more memory, but LLMs are best served from a headless server so that's what I'd recommend.

▲

mirekrusin 10 hours ago | parent | next [-]

Your agent(s) need to work on something, ie. running TypeScript, your app, your tests, Docker, Redis and/or database you need to run harness and user side apps ie. VScode, browser etc. it all adds up quickly.

Single user conversation spawns multiple parallel backend conversations, you need extra room for it as well, not just single context.

This plus usual apps like Mail, Spotify, iTerm2, SourceTree etc. also fill in memory.

Also 4 bit quantization is already quite aggressive compromise (measurable but sometimes acceptable loss, compared to ie. 8 bits which are often practically lossless) – for weights it's ok'ish, sometimes (especially if model was trained as 4 bit quants aware), but activations need to stay at higher bits taking more memory, otherwise quality degrades a lot.

For a dedicated headless setup, I’d probably use something like NVIDIA DGX Spark rather than a Mac (to be more precise NVIDIA GB10 Grace Blackwell from other suppliers than directly NVidia, they are much cheaper and have same insides). Linux is much better for running headless server, you also get standard NVIDIA/CUDA ecosystem instead of being tied to Metal/macOS.

For people who are interested in buying IMHO I'd wait a bit – next generation of Spark and/or Macs that are going to come out next year will be much better / will cross the line of being actually useful, not just a toy with goldfish LLM.

▲

hawk_ a day ago | parent | prev | next [-]

Is this for setup for agentic coding? Why not also run the IDE compiler etc... on the same machine to use those CPU cores as well?

	▲	0x457 a day ago \| parent \| next [-]
		Keep in mind that if you want MTP it adds a few gigs. If you use sub-agents it turns already slow generation into even slower generation. Won't be doing any compling (so rust, c and probably go are not avaiable) becase those add memory pressure during compiling. 32gb of unified memory is enough enough for system to be used for anything other than LLM generation.
	▲	redox99 a day ago \| parent \| prev [-]
		You can, you just need a beefier PC, and it's more annoying in terms of noise and heat vs throwing something on your server closet. Plus you don't need to worry about other software stealing resources and whatnot.

▲

qeternity a day ago | parent | prev [-]

> Edit: I'm thinking of a headless Mac mini, if you meant running it on the same machine you're using of course you'll need more memory, but LLMs are best served from a headless server so that's what I'd recommend.

What? LLMs are best served from a massive PD disaggregated cluster of B300s connected via NVLink.

If you're running LLMs on a Mac Mini, it's because you want to run local, not because it's the best setup.

	▲	redox99 10 hours ago \| parent [-]
		>massive PD disaggregated cluster of B300s connected via NVLink. So a headless server. Macs were mentioned because that's what the post is about. It could be a PC (I use a 2x3090 PC). The point is that it's a better experience to have a box dedicated to the LLM than running it in your system. Obviously in your home, so local.