One of the biggest limitations right now is memory capacity (storing large models/contexts in memory) and bandwidth (transferring the relevant data/weights to the silicon that is performing the operations on that data). This would cover things like:

1. having more memory on the card/chip and/or faster access to that memory;

2. integrated memory and compute units optimized for matrix and vector multiply add operations;

3. optimized load circuitry to e.g. read memory in the stride and span (next row, next column) access patterns common to matrices or ensure that no/few parts of the chip are stalled waiting on data or operations to complete.

Another aspect is quantizations. These are similar to SIMD vector operations in that you are performing an operation on a block of n-bit data values at the same time, so can have optimized circuitry.

For 2 or 3 valued quantizations you can reduce various addition and multiplication operations to logic operations, avoiding circuitry for things like the half-adder, full-adder, and carry-lookahead.

Then there's adding specific circuitry for common operations such as ReLU like is done in hardware acceleration of image, video, etc. processing. There's a trade off here as optimized hardware would perform better at the specific operations but if those are too specific then they can't be used by different/newer model architectures. (Though it does make sense to try and optimize common operations/logic where possible.)

It would be interesting to see if these designs can/will benefit training as well, as that would bring down the time/cost/energy of training large models as well as making it easier for local fine-tuning.

A large fraction of the innovation in CPUs is driven by working around the memory wall. I anticipate AI inference will follow the same trend, and innovations that work around the autoregressive nature will be enormously impactful.

Speculative decoding is an example. An accurate draft model can reduce the number of times you stream through memory by a factor of 4x.

> The list of individual innovations that led to today’s ubiquitous, powerful personal compute could fill dozens of books.

https://a.co/d/0dMk8urP

which is the Hennesey and Patterson computer architecture book would serve the role of the "dozens of books" hyperbole rather well.

The analogy is flawed - in Scrabble you get rewarded for unlikely word combos. Also causal attention attends to prior tokens (tiles already placed).

Big money from taking Tesla's place in line for an Nvidia shipment during a huge shortage without compensation.