This point gets raised every single time managed languages and low latency development come up together. The trade off is running "fast" all of the time, even when you don't have to, vs running slow most of the time and tinkering when you need to go fast.

I've spent a fair few years developing lowish (10-20us wire to wire) latency trading systems and the majority of the code does not need to go fast. It's just wasted effort, a debugging headache, and technical debt. So the natural trade off is a bit of pain to make the hot path fast through spans, unsafe code, pre-allocated object pools, etc and in return you get to use a safe and easy programming language everywhere else.

In C# low latency dev is not even that painful, as there are a lot of tools available specifically for this purpose by the runtime.