> But every optimization you make will allow you to run more inference with this hardware

Unless the optimization relies in part on a different hardware architecture, and is no more efficient than current techniques on existing hardware.

> there's not a reason for it to make it meaningless any more than more efficient cars didn't obsolete roads

Rail cars are pretty darned efficient, but they don’t really work on roads made for the other kind.