I'm from the times when you had to purchase a separate chip to perform floating point math. It was called a math co-processor. [1]

After a few generations (and over a decade) that was indistinguishable from the CPU chip itself.

It's a long hyperbole, I know, but I think local inference is inevitable; and the big fishes know it.

Will that be a complex technical setup? An appliance? An additional chip in your motherboard? So transparent it's burned right into the CPU? Those are just implementation details. We're probably just one generational breakthrough away from it.

[1] https://en.wikipedia.org/wiki/X87

Like the math co-processor it might end up just being new instructions for the cpu to handle ai related math.