Most of that power usage is moving data and weights into multiply accumulate hardware, then moving the data out. The actual computation is a fairly small fraction of the power consumed.

It's quite likely that an order of magnitude improvement can be had. This is an enormous incentive signal for someone to follow.