Most of the energy consumption comes from moving data between memory and compute units rather than the math itself. While HBM stacked on a silicon interposer was a signification efficiency improvement over traditional DDR, there's still room for improvement. The high energy costs will drive this further, e.g. near-memory/in-memory compute (PIM) or tighter 3D packaging and possibly optical interconnects. (And of course training/model optimisations)