this is horrifying, and contrary to the very notion of what a "driver" should be

To be fair the graphics APIs are provided as libraries with as much as possible done in userspace. Sandboxing that without any coupling at all would likely require either new kernel APIs or highly questionable virtual memory shenanigans.