This is sort of a solved problem with virtio native context. You can just run mesa driver in the virtual machine and proxy it's ioctl calls to host kernel gpu driver.

I'm pretty sure all the major GPU drivers in mesa now supports this feature.

Wonder why QubesOS hasn't adopted this yet.

They use Xens pci passthrough, you can setup a pci device to be assigned to a vm.

The vm runs it's own driver and has direct access to the device, you typically have to hide the device from dom0 for it to work flawlessly and hardware support is required. But I believe the kernels do not interact with each other in a direct way.

I used to have crashes on trying to restart a vm with a pci device attached probably related to resetting the device. But they seem to have resolved now days.