I suspect an AI could design a purpose-built FPGA-like replacement that would be, for its purpose, significantly more effective than the current general-purpose FPGAs.

It could have a small improvement on power consumption, but the current design can already achieve 90% of the maximum theoretical speed of this hardware without giving up programability/flexibility

FPGA is not efficient in any way except producing very few pieces of a particular logic very quickly.

But I think the higher level logic could check out. If we can abstract the majority of the compute into dedicated ASIC then we might use something similar to FPGA glue to connect and reconfigure them to adapt to new model updates. A bit similar to LoRA layers that you find tune to adapt the model to your particular needs.

I assume that it starts making sense once you have big multi year contacts to run a particular model with only minor updates.