I am not a proper developer and only use AI for faster research of topics so please forgive my ignorance. Could one not save a lot of money on tokens by using the 80/20 or 90/10 rule in that 90% of AI usage is on local models and save that last 10% or less for the frontier models where the local model did not meet the needs? Did they cover this and I misunderstood?

▲

jack_pp 2 days ago | parent | next [-]

Unlees you invest thousands your local ai won't even come close to the cheap cloud llms. Local is only worth it if you care about privacy or have a legitimate usage for the hardware otherwise. Money wise it isn't worth it

▲

Bender 2 days ago | parent | next [-]

I think you bring up a good point about privacy. Companies probably should not be handing over intellectual property to cloud AI third party vendors. Perhaps all of that should stay on-prem and then the cloud vendors could be used to fill in the gaps.

	▲	alchemism 16 hours ago \| parent [-]
		Enterprises running in public clouds have Zero Data Retention policies with the vendor. Routing the ai spend through Azure or AWS usually bring these policies along with it, ie enterprise data and useage is not ingested to retrain the model.

▲

nicman23 a day ago | parent | prev | next [-]

the qwen 27b is quite good for doing a mvp

▲

arijun a day ago | parent | prev [-]

I mean, if you're using one of the $200/month plans, you'd recoup the cost in a year or two.

	▲	TOMDM a day ago \| parent [-]
		You're not running an Opus 4.8/GPT 5.6 tier model on $5k of hardware at a useful Tok/s

▲

hotstickyballs 2 days ago | parent | prev | next [-]

This is like the "half of my marketing spend is wasted" quote. The complexity is finding out which half.

▲

Bender 2 days ago | parent | next [-]

I think the idea or methodology is something along the line of one starts with the local models then when they hit a wall then continue with their current results in a frontier model, thus the other half is those last bits one could not compute locally.

	▲	willis936 a day ago \| parent [-]
		If you are only using chatbots for code gen then for sure this is a reasonable approach. If you're trying to learn something then your best bet is to not have the lowest jpg quality version of human knowledge be your teacher.

▲

willis936 a day ago | parent | prev [-]

We could probably get the same results by offloading 20% of obviously smaller / simpler / well defined tasks to smaller models and keep the 80% for ones that benefit from the big iron. Not the 80/20 they were referring to, but still it's something.

I find myself wasting time on smaller models or wasting money on frontier models. I only have so much of either.

▲

pianopatrick 2 days ago | parent | prev | next [-]

To run a local AI that is half decent at research at usable speeds requires hardware that costs thousands of dollars.

Spending thousands of dollars on hardware to save dollars per month on tokens does not make financial sense.

If you run the numbers you'll probably find that using cheaper cloud models makes more financial sense than running those same models locally.

▲

Bender 2 days ago | parent [-]

I suppose I was thinking in terms of companies that already had a slew of hardware and are just missing external GPU's, FPGA's or whatever people are using these days. A company might have a $50MM budget for tokens but could save a chunk of that using their existing hardware with some modifications.

▲

pianopatrick 2 days ago | parent [-]

Maybe that makes sense if the company already bought the hardware.

But I currently think the fundamental reason the cloud models often make more sense is based in how the models work. My current understanding is that once these models are loaded into GPU memory, the model can respond to multiple prompts in parallel. So a cloud vendor that receives lots of user prompts at the same time can achieve really good hardware utilization. These cloud vendors can then spread the cost of the hardware over many more users than you can with a local model, leading to lower prices per prompt in the cloud.

	▲	graemep 2 days ago \| parent [-]
		If you are an individual, almost certainly, unless you are using LLMs very heavily. If you are a business employing many people all of who use LLMs reasonably heavily? You can still spread the hardware cost over many users and respond to multiple prompts in parallel. At some point buying the hardware will make sense. I would not get a dedicated server to run a single small VPS. On the other hand if I needed a lot of capacity a dedi is cheaper than a big VPS (or lots of small ones). There is also a benefit for anyone who has confidential or secret information.

▲

bkotrys 2 days ago | parent | prev [-]

That can work, especially for privacy or repetitive tasks. My experiment focused on subscriptions I already had, but local models are a natural addition to the routing layer.