This whole series is fantastic! Does an excellent job of explaining the theoretical limits to running modern AI workloads and explains the architecture and techniques (in particular methods of parallelism) you can use.

Yes it's all TPU focussed (other than this most recent part) but a lot of what it discusses are generally principles you can apply elsewhere (or easy enough to see how you could generalise them).