This still greatly affects connections/second, which is an important metric. Especially since servers don't always like very long lived connections, so you may get plenty of connections during an HTTP interaction.

It doesn't "greatly" affect it at all. The extra traffic and time required between curve25519 and ML-KEM768+X25519 is actually less than the jump from RSA2048 to RSA4096. Imagine how silly a person would appear if they had been this alarmist about RSA4096. When building for scales where it may eventually add up you should already be taking such scale into consideration.