These are going to be so awesome to buy in 5-8 years!

Right now I’m running 2x 3090 and 128 gb ram on 3.8 flash-next180b (the quant is 97gb) and it hums at120t/s. It’s really a surprisingly good model for general tasks, and I’m soon to see how it does with more agentic kit. For agentic work it does slow down quite a bit when the context fills up… but what do you want for running a 180b model on consumer hardware?

Right now I have it set up as an administrator for our asterisk PBX and radius server, and it does higher level judgment on our cctv footage for notifications to security. It’s been really good so far. Agents are the new UI, everything just works over SSH now.

▲

mnadkvlb 30 minutes ago | parent | next [-]

How do you do the cctv analysis ?

▲

locknitpicker 2 hours ago | parent | prev | next [-]

> (...) it hums at120t/s. It’s really a surprisingly good model for general tasks, and I’m soon to see how it does with more agentic kit.

This is exactly why I interpret the aggressive price cutting by OpenAI on their Luna model line as a clear sign they are scared shitless of local models, and how they can trigger a "the king has no clothes" moment for AI companies like OpenAI.

▲

Aurornis an hour ago | parent | next [-]

> as a clear sign they are scared shitless of local models,

The number of people with hardware to run a model like that is very small.

It's going to be many years before hardware to run medium size local models is widely available. Running mid-size local models doesn't even register as a factor in pricing decisions right now.

It's such a small part of the market and will be for many years, until RAM and other hardware prices come down.

	▲	stymaar 16 minutes ago \| parent \| next [-]
		> The number of people with hardware to run a model like that is very small. True, but the number of people who pay the non-subsidized API price is also very small, and it's all in big companies who would have no issue buying or renting hardware for that purpose.
	▲	locknitpicker an hour ago \| parent \| prev [-]
		> The number of people with hardware to run a model like that is very small. That's not the problem anyone is facing, and analysis anyone is doing. At all. The problem is the huge AI-related budget that companies who dove head first onto AI are facing. Therefore the problem everyone is faced with boils down to whether they keep paying the likes of OpenAI a de-facto business tax, or they replace that tax with a one-off CapEx invoice followed by a utilities bill. And you don't need a frontier model to do most things. In fact, the biggest challenge in using local models in agentic coding is the models supporting these agent-based tools. > It's going to be many years before hardware to run medium size local models is widely available. Nonsense. Speaking for myself, my employer already introduced a hard cap on AI credits while rolling out hardware support for local models, and even went to the extents of rolling out workstations with 64GB of RAM as a baseline. Consequently, we are using the likes of gemma4 for a fair share of everyday tasks. This has a transformative effect on your workflows because suddenly you don't ration tokens, and agent orchestration doesn't become a cash guzzler. So far the only constraints are tool support and token throughput, but that's not a concern.

▲

vladgur 2 hours ago | parent | prev [-]

At cost of power at $.45kwh here and dual 3090 setup easily approaching 1kw per hour at 120 tok/sec, latest Luna is cheaper per million tokens.

Your electrical costs are likely less than mine though

	▲	K0balt an hour ago \| parent \| next [-]
		Yeah electric costs are nil on the microgrid but capacity is limited.
	▲	selicos an hour ago \| parent \| prev [-]
		I'm working off a 128gb Strix system on mostly hydro and it's worth it for me. Qwen 3.8 27B and Flash Next, Gemma 4 and MOE models. It Keeps my thinking and methods local while moving at a speed I can actually read and think about. It can be annoying when you're running something long and realize there is an issue. But hey, I'm not wasting paid tokens. Reset and try again. I've probably thought of a better way by then too.

▲

sixothree an hour ago | parent | prev | next [-]

Are you running club-3090 or something else?

	▲	K0balt an hour ago \| parent [-]
		Flash-Next runs on Strata, version 0.1.42, an inference engine built specifically for Qwen3.8-flash-next

▲

x0z 4 hours ago | parent | prev [-]

> These are going to be so awesome to buy in 5-8 years!

Literally my first thought!