> I suspect the economics favor centralized servers, if you only look at the aggregated cost to serve X number of users' tokens.

The economics of real-time, low-latency inference of very large near-SOTA models will heavily favor a centralized setup. But if you can afford to wait for your answer - be it a day, a week, or even more at the extreme low end (or if you just stick to leaner models for your relatively quick replies) the economics start to shift in a very clear way. A slow-going local inference setup relying on cheap SSD offload does not need the high power input of a datacenter rack, and the cooling load is outright trivial - even when working on many requests in parallel, which (in a SSD offload context) is what maximizes throughput even for local inference. These are serious problems for centralized inference that will probably limit the scale at which it can be applied.

iPhone are in practice more reliant on central servers than ever before. Except for some games, if you take a random person's iPhone it becomes almost useless without internet connection. Which is exactly my point, it's not about compute, it's about people's expecations and priorities. The main driver of the need for higher performance chips on iPhones is its camera and smooth 120fps UI animations, not local processing for apps for almost all people.

However, the compute situation just skew it even further when it comes to LLMs.

A lot of the compute that iPhones do either has high bandwidth requirements (photo editing etc) or low latency requirements (games etc). LLMs are an interesting case where unless you're feeding in images, it's just text

Could it be some sort of permanently routed MoE where they detect and switch for the whole prompt instead of token by token?

Yeah but here you describing the opposite phenomenon. You're saying that the hardware is going to become cheaper and more powerful with the years, to the point a current State of the Art model from today will run on a normal consumer hardware in ten years. What people are trying to do now is the opposite, optimize the software as much as possible so that it does not need the best hardware but the normal one we currently have. As if we were trying to make a current AAA game to run smoothly on the 1080 GTX of your example.

Ok, now do memory capacity and bandwidth - the things that actually constraint local LLMs.

Speaking of ASICs - how likely is it that as models get better we'll see someone baking a whole model directly into the silicon? It's like having l0 cache.

Isn't there such a thing as low hanging fruit?

Aren't we already approaching theoretical physical limits? We're at 2nm

Sadly while the FLOPS are increasing nicely, total graphics memory is stalled in consumer cards by comparison.

We will see such power and price now only when AI market crashes or China reaches node parity and goes after market share as currently the way they are buying out most of the latest node production the consumer prices will only be palatable to the very rich or we will need to be happy with older slower nodes

Just a note that I think the direction most people are paying attention to is memory bandwidth; thats the real bottleneck and “number go up” but also constraint people are designing around

Except the 499$ of a 1080 GTX inflation-adjusted only buys you a 5070 or 5070 Ti even by MSRP.