To me, it rather meant: "Ultrahard" optimization is perfectly fine and a good idea, but not before it has become clear that the requirements won't change anymore (because highly optimized code is very often much harder to change to include additional requirements).

Any different interpretation in my opinion leads to slow, overbloated software.

Yeah - I've heard that described as "It's easier to make working things fast than fast thing work" - or something like that.