Remix.run Logo
▲ herf 3 hours ago

I love local AI, but even so, I admit the economics point the other way. Unlike gaming, AI is a uniquely good match for data centers because the applications today aren't latency-sensitive at all (except perhaps DLSS/self-driving/robotics). Most importantly, utilization is massively better with always-on servers doing parallelism (in-flight batching). Also in some areas (like California) our local electricity is quite expensive, so moving the work 100ms away can help. The privacy implications are horrifying, though.

▲younglunaman 3 hours ago | parent | next [-]

I tend to disagree here. Think about how useful it would be to have a nice llm in real-time debugging your applications loaded entirely on your computer. That sounds pretty sweet to me. I don't see how that's not really cool / immediately useful to you?

Maybe your point is the majority of the time that's not needed, fair, then maybe the approach is both?

▲herf an hour ago | parent [-]

Well, LLMs handle raw data perhaps a million times slower than conventional algorithms, so in "bytes" and "latency" terms, a dialup modem can handle most AI communication today - a rack of cloud GPUs 100ms away is still faster for a lot of things than $10k of hardware in your GPU, Mac, etc.

▲drdexebtjl 3 hours ago | parent | prev | next [-]

> the applications today aren't latency-sensitive at all

This is circular reasoning. We don’t have latency-sensitive applications because sufficiently powerful local AI is not profitable to entertain such applications.

▲herf an hour ago | parent [-]

Absolutely, and there are applications like self-driving or robotics or surgery or whatever where it's critical to have the work be local and never have a network hiccup. But these are all in some sense new markets, though I guess these would be growing faster if we had abundant local AI. Right now most of the AI applications that people pay for are ones that make human workers more efficient, and humans are already high latency.

▲musha68k 3 hours ago | parent | prev [-]

The economics point the other way because capital is aligned against you. Should this be a bubble that pops we will hopefully see prices come down again.

MooresLawIsDead and his spidey web / senses thinks it already popped:

https://youtu.be/YJeK2gumpEI