Most of the graphic drivers is already in user space. What's in the kernel is there the to manage memory and handle coordinate command submission and scheduling as well as display output configuration. There is no reason why this shouldn't be in the kernel.

But why do we need 499MB for such simple tasks, where all the rest of the kernel combined still fits in 135MB, with all the drivers for all the other devices, including Intel graphics?

I mean, it's 3.7x larger than all the rest of the kernel (135x3.7=499), how is that in any way reasonable for such a small set of functions across so few devices to take up so much space?