I don't really understand your setup, but if you throw enough money at ATEN they will be happy to give you a KVM that handles 4K @ 240Hz. Supporting high refresh rates is the main thing that would keep me from using most KVMs for gaming.

Ah, I neglected to mention that my keyboard isn't separate hardware. I'm using the built-in keyboard and display of my laptop, and would like to continue doing so. So hardware switches that depend on dumb monitors and keyboards don't work for me (and that's why I said KVM, not switch).