Economies of scale.

You need a cluster of 8-12 H100s to run the largest models locally.

It doesn't make sense to run these locally yet unless your use case also involves making it available for several dozen concurrent users.

▲

kingleopold a day ago | parent [-]

not to miss, future models will be more compute hungry too. Current hardware prices are still goin up and no it's not cheaper to run your AI for like %99 of the people because of lots of costs, it's not just hardware.

▲

lardosaurusrex a day ago | parent | next [-]

I think this fails to take into account how many people are fine with "fast enough" vs "fastest".

I've seen people happily use AI that takes several minutes to generate text or edit an image because to them they already aren't using their computer when they tell it to start; they just grab their phone and walk away and come back only to check in on it.

I feel like people here and on other technology discussions -- although it's worse here -- don't seem to parse what being the minority means.

They know they're one of the few to have access to such incredible hardware -- whether it be rented or purchased for way too much cash -- but they only see their own kin; their own ilk. They only compare themselves to the best.

The reality is that nobody expects data centre speed nor power in their own home and are satisfied to just go "haha its thinking" and let their computer quietly tick in the background as opposed to paying outragious prices for subscriptions or hardware.

▲

gmadsen 19 hours ago | parent [-]

That is completely discounting future capabilities and new use cases. Sure in 10 years you will have current SOTA locally, but in no way is it obvious we are anywhere near the limit of marginal value from improved capability

	▲	lardosaurusrex 19 hours ago \| parent [-]
		Unless you're a developer or doing complicated research you do not need much to use an LLM at home with consumer-grade hardware. I'm editing photos with 32GB of DDR4 RAM at 3200mhz alongside a 3050 with a nearly decade-old mobo and ryzen 5800XT and at most? It takes 30 seconds and that's allowing the gpu to use my actual RAM as 'fallback' memory. There are photoshop filters -now- that take longer than that before the LLM craze even began. Hell; I can train a lora in an hour or two. This obsession with "more specs more data faster and faster" isn't going to win; it already isn't winning. Deal with it or get wrecked.

▲

ericd a day ago | parent | prev [-]

The per token costs plummet with more concurrents. A box that can do 100 tps at request depth 1 might be able to do 3000 tps at request depth 64. Less per thread, but massively more per GPU/joule/etc. That’s the economy of scale of running in a DC rather than locally that they were referring to.