Finally, a reason to own a 128GB Strix Halo or GB10 device. Or a reason to consider the new Mac Studio.

I have a Strix Halo and dual 32GB GPUs in my desktop, and the latter is pretty much always better for running local models because it's quite a bit faster due to higher memory bandwidth. There simply haven't been any models that are better than Qwen 27B or Gemma 31B, which run comfortably in 64GB with big context.

And, MoE should make it run at a close to usable speed.

▲

AbsurdCensor 5 minutes ago | parent | next [-]

I have had an impossible time getting 120B or better models running on Strix Halo (especially under Windows) with any large context windows. And 30-40 tokens/second is fine, but not the fastest.

For the most part lately I have been sticking with Qwen 3.8 27b and that thing will easily suck up 64gb of ram. Add in docker with some additional programs running and it's really easy to eat up 128gb of ram.

▲

sosodev 3 hours ago | parent | prev | next [-]

That’s only true if you think AI is the only reason to own a powerful and efficient server. Mine does plenty of traditional server stuff too.

▲

SwellJoe 3 hours ago | parent | next [-]

I can do traditional server stuff on any old computer with a big hard disk and a decent amount of RAM. That's not worth $3500-$4000. When RAMpocalypse is over and we can buy a Strix Halo for under $2000 again, the math starts mathing. It becomes a pretty great desktop computer that also happens to run AI pretty well at a pretty good price.

▲

sosodev 2 hours ago | parent | next [-]

Yeah, but that computer can’t also do the AI stuff. And not everybody has a desktop with multiple 32GB GPUs available.

I’ll admit though I’m biased because I bought my board for $1600 back before the prices went crazy.

	▲	downrightmike an hour ago \| parent [-]
		Oh no a tough constraint that will lead to further innovation like deepseek. How terrible.

▲

surcap526 26 minutes ago | parent | prev [-]

[dead]

▲

ArvidSu 3 hours ago | parent | prev | next [-]

An "AI" server can do traditional server stuff but a traditional server can't do AI stuff (inference)

▲

2 hours ago | parent | prev [-]

[deleted]

▲

jubilanti an hour ago | parent | prev | next [-]

A 3060ti 8gb, released in 2020, has 448 GB/s of bandwidth compared to the Halo 256 GB/s

The 3080ti is 912.4 GB/s

	▲	embedding-shape an hour ago \| parent \| next [-]
		And the newly announced/launched Apple M6 has 170GB/s of unified memory bandwidth, meanwhile M5 Ultra gets 1.2TB/s of unified memory bandwidth. https://www.apple.com/newsroom/2026/08/apple-introduces-m6-a... Not sure if the first one is a typo on their press release, can't be just 170GB/s then be pushed for AI use, can it? Could be a different measurement I suppose...
	▲	yjftsjthsd-h 23 minutes ago \| parent \| prev \| next [-]
		But it also has 8GB of RAM.
	▲	SwellJoe an hour ago \| parent \| prev [-]
		Enjoy Gemma 4 E2B at blistering speeds, I guess?

▲

throwaw12 3 hours ago | parent | prev [-]

how much performance (tok/s) can you expect from 128GB Strix Halo? assuming this model will be released with FP8

also can you use it for fine tuning?

▲

SwellJoe 2 hours ago | parent | next [-]

The Strix Halo and DGX Spark are pretty danged slow, relatively speaking. I don't recall exact numbers, but with MoE models in this size ballpark (Laguna S 2.1), I seem to recall I was seeing about 20-25 t/s with a big context, which is close to usable. Qwen 3.8 27B crawls on this hardware, though, at 10-16 t/s, definitely not comfortable for interactive use. (Though this makes it seem like you can cook pretty good with a 4-bit ROCmFP4 quantization: https://github.com/julianmb/q38rocm the model does get notably dumber below six bits.)

A model similar in size to Laguna S 2.1, but with only 6B active parameters, should be a notable amount faster, so I would imagine 25-30 t/s would be a reasonable guess for where Qwen 3.8 Flash Next will land.

DFlash2 might improve all these numbers. It wasn't available last I was testing new models on the Strix Halo; I've only used MTP (which doesn't generally improve MoE models, but I believe DFlash2 can).

Given software improvements, I'm hopeful an MoE in this size range will be the sweet spot that pushes past 40 t/s and is also smart enough for real work. Qwen 3.8 27B is finally a self-hostable model that's smart enough, but it thinks so hard it still isn't really useful for agentic interactive use.

Note also prefill with large models is pretty slow on the Strix Halo (300 t/s, maybe). Time to first token is a painful wait, when using it interactively with large models.

▲

downrightmike an hour ago | parent | prev [-]

You can only use up to 90gb for the GPU, so it doesn't fit

	▲	SwellJoe an hour ago \| parent [-]
		No. In Linux, you can ignore the BIOS (well, set it to 512MB) and set GTT to allow almost the entirety of memory for GPU. If you don't run a desktop GUI (and why would you), you don't need memory for much of anything else. Just a few GB for the OS and llama.cpp. These are roughly the settings I use: https://github.com/kyuz0/amd-strix-halo-toolboxes#kernel-par...