Funnily enough, "GPU" (video driver) on Linux used to be microkernel-like: the Xorg would run in userspace and talk to device directly. It was exactly as you described - if Xorg crashed, you could use keyboard shortcut to reinitialize it.

And the major motivators for KMS/DRM, which moved a lot of that functionality into kernel, was performance and stability.