And the newly announced/launched Apple M6 has 170GB/s of unified memory bandwidth, meanwhile M5 Ultra gets 1.2TB/s of unified memory bandwidth. https://www.apple.com/newsroom/2026/08/apple-introduces-m6-a... Not sure if the first one is a typo on their press release, can't be just 170GB/s then be pushed for AI use, can it? Could be a different measurement I suppose...

I can do traditional server stuff on any old computer with a big hard disk and a decent amount of RAM. That's not worth $3500-$4000. When RAMpocalypse is over and we can buy a Strix Halo for under $2000 again, the math starts mathing. It becomes a pretty great desktop computer that also happens to run AI pretty well at a pretty good price.

An "AI" server can do traditional server stuff but a traditional server can't do AI stuff (inference)

The Strix Halo and DGX Spark are pretty danged slow, relatively speaking. I don't recall exact numbers, but with MoE models in this size ballpark (Laguna S 2.1), I seem to recall I was seeing about 20-25 t/s with a big context, which is close to usable. Qwen 3.8 27B crawls on this hardware, though, at 10-16 t/s, definitely not comfortable for interactive use. (Though this makes it seem like you can cook pretty good with a 4-bit ROCmFP4 quantization: https://github.com/julianmb/q38rocm the model does get notably dumber below six bits.)

A model similar in size to Laguna S 2.1, but with only 6B active parameters, should be a notable amount faster, so I would imagine 25-30 t/s would be a reasonable guess for where Qwen 3.8 Flash Next will land.

DFlash2 might improve all these numbers. It wasn't available last I was testing new models on the Strix Halo; I've only used MTP (which doesn't generally improve MoE models, but I believe DFlash2 can).

Given software improvements, I'm hopeful an MoE in this size range will be the sweet spot that pushes past 40 t/s and is also smart enough for real work. Qwen 3.8 27B is finally a self-hostable model that's smart enough, but it thinks so hard it still isn't really useful for agentic interactive use.

Note also prefill with large models is pretty slow on the Strix Halo (300 t/s, maybe). Time to first token is a painful wait, when using it interactively with large models.

You can only use up to 90gb for the GPU, so it doesn't fit

I built a router that lets you route between local and cloud models. Link in my profile.

I haven't yet, though plan to when this model is released. The models that I've been daily driving (Gemma 4 26B, Qwen 3.8 27B) have fit nicely on my 3090. I think FreeToken only offers a perf increase for MoE models that you can't feasibly fit in VRAM.

Yeah I'm sometimes unsure how to get best perf out of llama.cpp, and honestly thought it already did what the FreeToken paper discusses, but from everything I've been able to find since llama.cpp has no dynamic expert cache for GPU. An RFC discusses adding such capability and there's impressive results some are claiming from a fork, but I had to stop reading the thread, reading all the LLM generated comments and summaries from people was making me dizzy.

RFC here https://github.com/ggml-org/llama.cpp/discussions/24528 which also links to some experimental implementations throughout the thread.

> It is still slow, a lot slower than what you are used to with claude and co.

That really depends on the model, I run a few models locally. All at speeds comparable to or faster than Opus.

In general we haven't reached the ceiling for what performance we can get out of consumer hardware. As evidence by FreeToken which hasn't even added MTP/speculative drafting support yet, which will add another boost.

> Then when it runs for 30 minutes for something claude needs 5, your device will get hot.

I doubt the timing differential here, but even still I run my 3090 pretty heavily with inference workloads and it stays cooler than when I use it for gaming.

> And even a used 3090 is apparently now between 1-2k.

Yeah I guess the price went up significantly in the last couple months, used to be hovering around 1k. 3090 isn't the only option though.

yeah, but otoh... f* Anthropic and OpenAI

Have you given Ornith-1.5-35B a shot?

It's been a pretty decent step up for me compared to Qwen3.6

https://news.ycombinator.com/item?id=49362401

What sort of pp/tg speed do you get on a Strix Halo?

it'll hopefully improve with more MoE and half the prefill/generation. I think it's the sweet spot for the strix halo for smarter or vibe tasks.

This is the way.

I’m doing that. Mac Mini M4 Pro with 48G RAM as a headless llama.cpp server.

I much prefer using " thin clients " as the interface to the big VMs running in my homelab

50-70tk/s is what I get on my m5 max on a 5-6bit Qwen 3.8 27B?

That was mainly before the M4 generation when they didn't have matmul instructions.

They normally release a 35b dense and an 27b moe (4B active per token)

For context 35B on my m4 runs at 10 tokens a second, 27B moe runs 50-60 tokens a second.

Good point, I didn't see that. I guess I categorized them in the same bucket of 'runs on 128gb machines'

Guess Qwen 4 is the one to wait for.

Qwen3.8/Qwen3.6 has a weird self doubt/thinking too much problem. You can prompt it away. I would say it "approximates" Opus 4.X class models well enough especially for coding/linux problems.

The only reason I stopped using it as much is I was getting 25-35tok/s on Intel B70 (non-quant) which made some responses slow. For a long running/autonomous task, it would probably be sufficient.

There is the rule of thumb that if you take the geometric mean of the total and active parameters of an MoE model you get the equivalent size of an equally capable dense model.

If you follow that formula, you would expect a 125b-a6b model to match a 27b model (sqrt(125*6) = 27.3). That does not feel like a coincidence

Thanks. My current stack ranking of anthropic models is:

4.6 ~= 4.8

4.7 much worse.

Fable and newer consistently tells me to pound sand, so I’m not sure what I’m paying $200/month for. 4.8 sometimes does too, but it’s at least usable most of the time.

So, I’d expect this to mostly replace Claude for my workflows. The main tradeoff for me should mostly be token throughput vs. no longer really trusting anthropic.

I've got the A10B hooked up to deer-flow and it does remarkable well when you dont need to baby sit it.

I’ll give it a try! Thanks for the heads up!

Not exactly surprising given it's a dense model at 2.7x the size of the experts in gpt-oss

The 27B one is great on a 5090.

This one is basically aimed at macs, Strix halo and DGX Spark.

You can run the 27B released last week. I haven't tried it yet myself but the 3.6 version runs great on my 5090.

Very sad. I try not to go below q8.