> though I think in practice network latency dominates so much that this kind of optimization is fairly low impact.

In practice, no.

Network latency is the least problematic part of the stack. I consistently get <3ms. It's largely encode/decode time which in my setup sits at around 20ms meaning any work in this area would actually have a a HUGE impact.