You just described X11's DRI (direct rendering infrastructure).

https://en.m.wikipedia.org/wiki/Direct_Rendering_Infrastruct...