The congestion control algorithm in TCP has some interesting effects on throughput that a lot of developers aren’t aware of.

For example, sending some data on a fresh TCP connection is slow, and the “ramp up time” to the bandwidth of the network is almost entirely determined by the latency.

Amazing speed ups can be achieved in a data centre network by shaving microseconds off the round trip time!

Similarly, many (all?) TCP stacks count segments, not bytes, when determining this ramp up rate. This means that jumbo frames can provide 6x the bandwidth during this period!

If you read about the network design of AWS, they put a lot of effort into low switching latency and enabling jumbo frames.

The real pros do this kind of network tuning, everyone else wonders why they don’t get anywhere near 10 Gbps through a 10 Gbps link.