At 6 out of 8 companies I've worked at (mostly a mixture of tech & finance) I have always had to fight to get any time allotted for performance optimization, to the point where I would usually just do it myself under the radar. Even at companies that measured latency and claimed it was important, it would usually take a backseat to adding more features.

That is how it is most of the time. If you want to experience the other extreme, go to HFT or low-latency projects.