You also pay for the increased throughput with significant memory overhead, in addition to worst-case latency.

This. The memory overhead kills you in large systems/OS-level GC. Reducing the working set size really matters in a complex system to keep things performant, and GC vastly expands the working set.

In the best cases, you’re losing a huge amount of performance vs. an equivalent non-GC system. In the worst, it affects interactive UI performance with multi-second stalls (a suitably modern GC shouldn’t do this, though).