Large data centers never made sense. If an additional pc with a graphics card was put into every home, we wouldn't need power grid upgrades. Our power grid is designed around human distribution i.e. homes. Large data centers cherry pick where it may be possible to quickly connect power connections up. Data centers need to secure their power connections before the utility rate hikes on everyone else generate public opposition.

If large data centers cannot secure free power connections paid for by everyone else, they cannot make any of this profitable. The whole thing was a scam from the start. This is why cloud AI companies bought up so much pc and server hardware. Artificial scarcity was the only way to slow down local llm adoption and sell the false narrative that cloud AI has value.

Plucking a graphics card out of every home and centralizing the cards to a handful of large data centers that require a trillion dollars in power grid upgrades is an idea so dumb, we should all be laughing at it. They even have to use active cooling or evaporate drinking water instead of free air cooling. The consumer doesn't benefit in any way from this. Consumers would essentially be paying more just to be datamined.

LLMs are not a bubble, large data centers are a bubble. Look at how large data centers are now the first target in any war. A graphics card in every home, doesn't give someone a concentrated target. National security makes large data centers extremely bad.

▲

Tuna-Fish 7 hours ago | parent | next [-]

Large data centers are dramatically more efficient at utilizing both the power and the hardware.

At it's core, running a top-tier LLM means linear load of trillion+ weights from RAM for every token. The cost of this operation absolutely dwarfs the cost of doing the actual math. If you are only serving a single user, you can only amortize this over a handful of token streams, times like 2-4 for speculative decoding. If you are running in a datacenter, you can batch thousands of streams, and get correspondingly much more tokens out of the same expenditure of energy and hardware.

▲

amluto 6 hours ago | parent [-]

Be slightly careful with this. Large datacenters use power to operate the facility (this is measured as PUE > 1). Datacenter-oriented hardware often contains low-diameter, high static pressure fans that consume quite impressive amounts of power that are not counted as overhead in the PUE calculation.

A home- or office-oriented workstation will have much less power-hungry fans (go compare a Noctua datasheet to a server fan datasheet - the Noctua wipes the floor with a server fan if measured by CFM/W, and the internal temperature rise is mostly controlled by airflow and not by static pressure). And in some climates and times of year the marginal facility HVAC cost associated with running a computer may be zero or even negative.

And lower utilization hardware may run in on a more efficient point on the power-vs-performance curve. [0]

That being said, your overall point is correct.

[0] For a dramatic demonstration of how much this curve matters, you can run a non-CPU-saturating, non-latency-sensitive workload on various Intel CPUs with and without turbo on. Seeing a 2x or greater difference in reported power is quite possible. (Intel is terribly at writing platform firmware and the default behavior may be wildly suboptimal.)

	▲	Tuna-Fish 6 hours ago \| parent [-]
		None of that matters when there is literally a 100x multiplier on efficiency because of batching. It's all a rounding error.

▲

orangecat 5 hours ago | parent | prev | next [-]

In addition to the batching issue, "a graphics card" is not going to cut it for the multi-trillion parameter frontier models. You're looking at a 512GB Mac Studio to support anything close to those model sizes, and that's with heavy quantization and much worse performance.

▲

eluru 4 hours ago | parent | prev [-]

agreed, gets some mid range hardware, some click installing agents with descent customisability, and im talking boring customisability like 'help me plan what to buy this month for my home' kind of tasks. Thats what we are going towards