I was already rolling around the idea of a 128GB M5 Max MBP. Now this!

A 4-bit MLX quant with 128k window should fit perfectly, in the 50-70 tok/s range.

Eric_WVGG 3 hours ago | parent | next [-]

Just out of curiosity, why run "local-local" when you could just set up a Mini or Studio at home and query it over http? [edit] whole conversation about this in another thread https://news.ycombinator.com/item?id=49433413

I’m personally considering retiring my MBP for a Studio + 15" Air whenever this MBP ages out.

	▲	kamranjon 23 minutes ago \| parent \| next [-]
		I actually do this with my MBP - it's a LLM server when I'm working - and then when I'm not it's just a really great machine for video editing and other media work.
	▲	LeBit 2 hours ago \| parent \| prev [-]
		This is the way. I’m doing that. Mac Mini M4 Pro with 48G RAM as a headless llama.cpp server. I much prefer using " thin clients " as the interface to the big VMs running in my homelab

▲

sscaryterry 5 hours ago | parent | prev | next [-]

I have a 128GB M5 Max, and it sucks at this stage. 50-70 tok/s might be something...

▲

smcleod 4 hours ago | parent [-]

50-70tk/s is what I get on my m5 max on a 5-6bit Qwen 3.8 27B?

▲

Casteil 4 hours ago | parent | next [-]

I don't know what black magic you're up to but I see more like 30-35t/s on a 16" M5 Max using 3.8:27b Q4, regardless of whether it's mlx or gguf.

qwen3.5:122b-a10b is significantly faster at around 60-65.

	▲	syntaxing 3 hours ago \| parent [-]
		With MTP? I get 25-30 TPS on a strix halo. 50+ on a M5 max should very doable. Dflash (2) will push your TG even further

▲

sscaryterry 3 hours ago | parent | prev [-]

I tried 8-bit, perhaps I should try 6-bit.

▲

irthomasthomas 4 hours ago | parent | prev | next [-]

IDK, prefill speed is a bigger concern for most wokflows, like agent coding, and I heard that this is quite low on macs?

▲

smcleod 4 hours ago | parent [-]

That was mainly before the M4 generation when they didn't have matmul instructions.

▲

jasonjmcghee 4 hours ago | parent [-]

M5 prefill is much faster than M4.

I've seen benchmarks that show 4-5x faster of M5 Max vs. M4 Max.

For local models you're likely using M5 Max, prefill is low thousands of tokens per second, as opposed to, say high hundreds with M4 Max.

For larger dense models, some fraction of that, but similar multiple.

	▲	smcleod 4 hours ago \| parent [-]
		Yes, I have the M5 Max. But there was no matmul acceleration before the M4 which made things a lot slower.

▲

3 hours ago | parent | prev [-]

[deleted]