The hoop I recently jumped through:

There's a type of input called "DeviceEvent" which is a bit lower level than "Window event". It also occurs even if the window isn't "active".

Windows and X11 support this, but Wayland doesn't except for mouse movement. I noticed my program stopped working on Linux after I updated it. Ended up switching to Window Events, but still kind of irritating.