Xorg is indeed a lot of painful complexity. This being said, the software is not Linux specific, and for modern Linux distributions, it is more and more a legacy technology.