I wonder if this is the same reason why Microsoft's Remote SSH plugin on VS Code is so flaky even with a decent internet connection. Every couple of months I try to give it another go and give up due to the poor keyboard latency I inevitably experience. And the slow reconnects whenever I glance away from my computer monitor briefly. This is on a fiber connection with a 20ms ping to the remote machine.

You surely mean the latency in its embedded terminal and not the code editor, right? I use VSCode’s remote SSH specifically so that code editing doesn’t suck. It really does not.