Why? GPUs are replaced every 3 to 5 years. This is going to be an ongoing operational cost forever. It will probably increase more if larger models require bigger VRAM sizes.

▲

mcbuilder a day ago | parent | next [-]

We have probably hit a limit to scaling LLMs through raw parameter count alone, at least we're not seeing the exponential pace. I personally think we'll end up with a nice sigmoid curve plateauing in the sub 10T parameter regime. The amount of tokens processed (in inference) is increasing exponentially though (I've been following open router usage stats for years and it's always been exponential). We will of course make technological advances in hardware efficiency, and model parameter efficiency, but I think a much more plausible future is that VRAM needed for loading and serving individual models will slow down or even stop. We will need more chips, and more power, as demand continues to grow of course, but the operational lifetime of GPUs today will be a lot longer than the SoTA cards from 5 years ago.

	▲	CuriousSkeptic 19 hours ago \| parent [-]
		> We have probably hit a limit to scaling LLMs through raw parameter count alone, at least we're not seeing the exponential pace. Source?

▲

ody4242 a day ago | parent | prev | next [-]

They are building new datacenters for the AI demand, so around half of this CAPEX is not for the GPU-s, and those will not be replaced every 3-5 years.

Also, TPUv2 was introduced in 2018, and still not completely retired in all regions, from accounting pov, it has been written down to 0, but they are still working.

▲

lokar a day ago | parent [-]

The cost of the land, building, mechanical equipment, etc is a very small fraction of the total cost of a DC.

▲

ody4242 a day ago | parent [-]

The upfront cost of facility is ~30%, network infra 10-15%, land/utilities is small percentage, power could be significant for an AI DC. The servers are ~50-60% only.

	▲	lokar a day ago \| parent [-]
		That was not my experience, but that was pre-Covid

▲

benoau a day ago | parent | prev [-]

That cost has always been there and allowed for their lucrative margins. It's the upfront cost of building/populating their datacenters (many more than before) that is eating those margins.

	▲	minraws a day ago \| parent [-]
		I mean the issue is scaling, the worlds for cloud never kept getting bigger and bigger and compute scaling had stopped a while ago in the CPU space. With AI every new generation with both massive hardware and software stack changes from Nvidia makes prior chips extremely inefficient to run, basically we are comparing an ASIC industry to a general purpose compute industry where all work loads are the same shape and size and so on. Margins for ASIC based mining companies or ASIC solutions providers were never high, Optane and other weird solutions are niche and great for a specific category or moment in time, but they become obsolete pretty quickly. The fear is we don't know if this Capex can stop. The worst type of fear is if this Capex will stop then what? Someone is very overpriced in this market, the cloud companies, the hardware providers or both. I don't see how we reconcile this without a massive wave of repricing, ofc markets can stay irrational and we don't see the actual books but AI doesn't have so much revenue. Suddenly the AI token/cloud revenue won't 100x in a year or two... Especially when intelligence will continue to get cheaper, the margin compression is a massive risk. All the data centers for hyper scalers were a miniscule part of their story the real moat was the software layer on top otherwise Hetzner would be priced like an Amazon as well. Something is shaky with this market I don't know what it's very opaque even as an insider working on for big tech and startups. I have no clue who falls first and which bottleneck cracks but there is not enough revenue for tokens, we will see a strong 2-3x growth in the next few years, from here which is absurd, but it's not enough, not nearly enough. If the capex keeps high and increasing. Ofc they can stop the capex and the otherside gets repriced it's not like nvidia, micron and co aren't worth trillions.