> all the quirks of the graphics drivers

This should ideally be solved at the kernel level - and I would argue it is solved there. Linux has the DRM abstraction for this very reason. Wayland actually builds on top of this abstraction, while Xorg was sitting in a strange position, not squarely in userspace.