Copying how VNC does it is exactly how my first attempt would go. Seems odd to try something like Moonlight which is designed for low latency remote gameplay.