One of the big issues was latency.

Yes, but the real issue (IMO) is that something is causing an avalanche of some kind. You would much rather have a consistent 100ms increased latency for this application if it works much better for users with high loss, etc. Also, to be clear, this is basically just a memory cache. I doubt it would add any "real" latency like that.

The idea is that if the fancy system works well on connection A and works poorly on connection B, what are the differences and how can we modify the system so that A and B are the same from it's perspective.