Yea UDP is technically more performant, but then you need a crypto layer + reliable message delivery layer + bespoke client. Using a plain old SSH client is cool.

However, there are existing libraries for exactly this use case - see https://github.com/ValveSoftware/GameNetworkingSockets

I guess QUIC libraries would also work.

its not really a question of 'udp performs better'. in tcp we have to live to head-of-line blocking on losses and congestion control. if you don't care about receiving every packet, but only the most recent, then udp is a good choice.

running without congestion control means that you avoid slowstart. but at a certain rate you run into poorly defined 'fairness' issues where you can easily negatively impact other flows. past that point, you can actually self-interfere and cause excessive losses for yourself.

quic uses congestion control, but uses latency estimates and variance as a signal to back off. it still imposes an ordering on a per-stream basis. so it might not be ideal either.

sctp has a mode which supports reliable and unordered, which might be something to consider

so really - if you care about latency and have a different reliability model, its worth unpacking all these considerations and using them to select your transport layer or even consider writing a minimal one yourself