Wouldn't a little google maps style navigation solve latency mostly?

Project on to a sphere, crop a little bit, do onset of motions by rotating or moving in the sphere