I'm trying and failing to find value running a potential Qwen 3.8 27b dense model on a 16 core, 128 GB of ram, 2080ti box. Yes, the GPU yells for help, but the problem is that no math works to upgrade this machine even when pouring $200 in rent every month into the large model providers...

How are you all justifying economical use of these local models right now? What's the cost efficient way to do this and do better (even with models evolving over time and losing now vs later) than the big labs?

▲

aurareturn an hour ago | parent | next [-]

You will simply not get more value out of running a local model vs paying for a subscription/API from the cloud in 2026. There is no math that will make local models come out ahead in $/intelligence/token.*

The point of local models is privacy, offline use, and maybe no guard rails.

* Not talking about enterprises that buy DGX racks and host Chinese models for internal use.

	▲	AureliusMA an hour ago \| parent [-]
		Points are starting to be made in favor of value, to the contrary of what you are affirming. Specifically because the new open weights models lower the TCO of hardware in an environment where new open weights were previously thought to be a thing of the past.

▲

jbellis 23 minutes ago | parent | prev | next [-]

The only scenario is if you have enough work to do batch inference. Using a tiny fraction of GPU capacity to decode a single request at a time just doesn't make sense, as you say.

▲

dpc_01234 36 minutes ago | parent | prev | next [-]

There's no point, except if you want privacy and independence. I am playing with a personal assistant that checks my emails, calendars, sends me an agenda and maintains my TODO list. I am not sending such data outside.

▲

AureliusMA an hour ago | parent | prev [-]

If you can do inference on the CPU, drop the GPU : it should be faster.