It will be interesting to see the intersection of this with inference engines like FreeToken which improve distribution of work for MoE models across CPU/RAM and GPU/VRAM.

If all it takes for a competitive model to run locally at good speeds is a used 3090 and some DDR4, then we might be in for the year of local AI.

https://github.com/FlashML-org/FreeToken

▲

kamranjon 2 hours ago | parent | next [-]

Have you tried FreeToken yourself? I was hoping to find some benchmarks on their github but took a quick pass at their research paper and it seems they're showing ~2x performance on qwen 3.6 35b when compared to llama.cpp - but llama.cpp is so sprawling and has so many options I find that a difficult comparison.

	▲	notnullorvoid an hour ago \| parent [-]
		I haven't yet, though plan to when this model is released. The models that I've been daily driving (Gemma 4 26B, Qwen 3.8 27B) have fit nicely on my 3090. I think FreeToken only offers a perf increase for MoE models that you can't feasibly fit in VRAM. Yeah I'm sometimes unsure how to get best perf out of llama.cpp, and honestly thought it already did what the FreeToken paper discusses, but from everything I've been able to find since llama.cpp has no dynamic expert cache for GPU. An RFC discusses adding such capability and there's impressive results some are claiming from a fork, but I had to stop reading the thread, reading all the LLM generated comments and summaries from people was making me dizzy. RFC here https://github.com/ggml-org/llama.cpp/discussions/24528 which also links to some experimental implementations throughout the thread.

▲

Zylokloto 4 hours ago | parent | prev [-]

You can already run it locally its just not the same.

It is still slow, a lot slower than what you are used to with claude and co.

And as soon as you increase context size, your memory requirements jump.

Then when it runs for 30 minutes for something claude needs 5, your device will get hot.

And even a used 3090 is apparently now between 1-2k.

▲

notnullorvoid 3 hours ago | parent | next [-]

> It is still slow, a lot slower than what you are used to with claude and co.

That really depends on the model, I run a few models locally. All at speeds comparable to or faster than Opus.

In general we haven't reached the ceiling for what performance we can get out of consumer hardware. As evidence by FreeToken which hasn't even added MTP/speculative drafting support yet, which will add another boost.

> Then when it runs for 30 minutes for something claude needs 5, your device will get hot.

I doubt the timing differential here, but even still I run my 3090 pretty heavily with inference workloads and it stays cooler than when I use it for gaming.

> And even a used 3090 is apparently now between 1-2k.

Yeah I guess the price went up significantly in the last couple months, used to be hovering around 1k. 3090 isn't the only option though.

	▲	ekidd 30 minutes ago \| parent \| next [-]
		> That really depends on the model, I run a few models locally. All at speeds comparable to or faster than Opus. Yes, a lot of Qwen3.8 27B setups are actually quite snappy, as long as they fit 100% in VRAM. In my testing, I wouldn't go below 32GB of VRAM, though—you really want a 6-bit quant and 8-bit K/V quants minimum. I've seen too much weirdness out of 4-bit quants since Qwen3.8 shipped. I think it may be damaged more than 3.6 at similar levels of quantization? If hyperscalers hadn't bought up almost all the fast RAM production for the next several years, 32GB of VRAM would be tolerably cheap—a lot by "home PC" standards, but not terrible by "professional tools" standards. Sadly, the RAM market is amazingly ugly right now. > I doubt the timing differential here, but even still I run my 3090 pretty heavily with inference workloads and it stays cooler than when I use it for gaming. Yeah, running inference on a laptop is likely to run quite hot. But in an ATX case with decent cooling, it's generally a lower load than gaming. One handy tip: Many Nvidia GPUs (and some from other manufacturers) support power limits. For example, limit a 5090 to 400W instead of 600W, and it will run much cooler. You might lose 11% off your tokens/sec (depending on the exact card).
	▲	nitin7 an hour ago \| parent \| prev [-]
		Which models when run locally come close to Sol and Opus, from your experience? And which harness do you use?

▲

blahblaher 3 hours ago | parent | prev [-]

yeah, but otoh... f* Anthropic and OpenAI