I suspect the economics favor centralized servers, if you only look at the aggregated cost to serve X number of users' tokens. But we could say the same thing about a lot of the computation that iPhones do locally. They could have been much thinner clients, but instead they now have more compute power than desktops had when iPhones launched.

I don’t understand the downvotes. It’s perfectly reasonable to disagree, but the argument is laid out clearly. Why not engage with it instead?

I don't understand, if they are only using a subset of the tokens then it's a sparse model. What do you mean by dense?

8800 GTX in 2006. Cutting-edge, an insanely powered consumer card for the time. Theoretically around 0.3456 TFLOPS.

1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS.

5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today. Theoretically around 104.8 TFLOPS.

In the same timeframe mobile processor CPU's went from 0.001 TFLOPS, to today's Apple's A19 Pro chip which delivers 2.074 TFLOPS.

That's _without_ getting into ASIC's, or purpose-built hardware like Taalas's model on silicon HC1, or generic AI dies like what they're planning with HC2 or Cerebras, which will massively compress the timeline.

AMD already demonstrated 1T on strix halo clusters. << $10K at original MSRP.

We'd bought 4 x $11K Mac Studios at my college and via exo, we had Kimi K2.5 at 30 TPS.

Not too wild an idea!

Especially if you’re Apple and your expertise and reputation is in making privacy conscious devices and cloud services that are exceptionally user friendly.

Then you go to the AI labs and say, “We see you’ve spent a trillion dollars on that model… we’ll write you a check today to let us run it how we want.”

Skips the ugly horse race and massive risk while focusing on what they do better than anyone.

And they’re the only ones who have cultivated the trust with normal people to be able to say, “We now have the only AI service that can safely reference your personal info without giving it all to strangers.”

They could possibly end up the second biggest winners in this insane hype cycle… without having had to compete on making a sota model.

A lot of this will come from co-optimizing hardware and low level machine code for this specific use case… something apple is coincidently very good at.

Apple has worked very hard to make unified memory a feasible approach, and the benefits of that are pretty clear in apple silicon- that efficiency not only results in power and therefore thermal gains, but also in a significantly faster full loop per process: or a faster time to token. This is why even their single core mobile chips in the budget line Neo out perform PC processors with several times more threads and RAM[1]. Turns out, unified memory lets you have a whole lot more control over things like RAM bussing and core use for specific workflows.

Speculatively, a unified memory approach could also allow you to more easily integrate things like ReRAM to solve the current memory swapping bottleneck.

Let’s say a friend of mine works hardware at apple and works on exactly this… on device processing is the future I’m betting on.

[1] https://youtu.be/x26A28DoT-w?t=605

The models themselves have far from plateaued. Maybe someone finds a way to get a really capable model down to, say, 12GB of ram. Then we'd be in business.

I think a major factor is memory bandwidth. Apple has raised it steadily for each M series generation, and that hasn't plateaued.

Nvidia leads in bandwidth and specialized architecture, but local inference takes off when it's usably fast at much lower cost and power consumption.

It could be on software side too. OpenAI has certainly not plateaued.

I could definitely image Apple embedding a kind of LLM-optimized FPGA: slow to load (update) an LLM, but blazing fast at computing tokens.

Who needs memory when your model is set in silicon ?

I do wonder if there are some experts that are more likely to be hit. So if the normal optimised setup runs in 12GiB an you have 4GiB extra to spare, you could say “promote the most used X experts to this stable (old gen in GC parlance) region and don’t swap it out.

Maybe you could even do something like profiling and remember over multiple sessions (per project/workspace) what the most used agents are and load those up before hand.

There is a limit how much you can squeeze out of given hardware. Betting on it being closer to 100t/s than 6t/s is only that, a bet.

I'm pretty sure one can rent a GPU for a few minutes with the electricity cost of leaving an M5 overnight.

There's read disturb on SSDs, enough reads will eventually force controller to rewrite the cell and it's neighbours.

Practically if you're not streaming weights 24/7 from a full SSD, then it shouldn't be a problem.

Yeah, that's why most of these comments seem weird to me.

Yes for NAND, and I suppose nobody is using mechanical hard drives for this.