> how would a display server work in an ideal world

When designed by committee.

With conflicting interests.

And Veto Powers.