How is the latency for drawing? I am going to start doing this, but it makes me think, it would be nice to have a way to disable full refresh while drawing, and doing tablet-side drawing over the current VNC frame while the stream is paused, and asynchronously forwarding the input which will hopefully recreate the same drawing path on the server.

I don't know how to do what you are describing, but right now, the drawing is OK. it's not instantaneous but good enough for me, not creating artworks. just doodles and scribbles. I just don't look at the tablet screen for drawing. I only use it as an input because the refresh rate is low for this. but the VNC latency itself is not annoying (I would guesstimate ~0.2s), although I think there are ways to make VNC latency less.

The latency for drawing on the Boox Note Max is excellent. Best on the market or close to it.