A large fraction of the innovation in CPUs is driven by working around the memory wall. I anticipate AI inference will follow the same trend, and innovations that work around the autoregressive nature will be enormously impactful.

Speculative decoding is an example. An accurate draft model can reduce the number of times you stream through memory by a factor of 4x.