There are a bunch of legitimate issues with X.

For example there's tons of legacy cruft in there intended for working with hardware that hasn't been in use since circa 1992. Things like monochrome 3D displays with weird resolutions like 1200x240 and non-square pixels. Having that stuff in there makes supporting more modern hardware more difficult than it needs to be (and is also part of the reason behind why e.g eliminating tearing is very difficult), and it adds huge complexity to the codebase for no benefit on modern systems, which makes it much more difficult (but NOT impossible, as some love to claim) to maintain.

There's also the wayland fanboy's go-to criticism: there are also some security shortcomings in the protocol. You can find details on this shortcoming which I have never in 30 years seen exploited in the opening paragraphs of every pro-wayland article on the internet. (it is a legit shortcoming. There have been multiple suggestions on how to address it over the decades without starting over from scratch. xlibre is working on one of these)

But over the years I've slowly become more and more convinced that the biggest issue people have with X is that it's not shiny and new.

I'm expecting them to announce a rewrite in rust any day now ;)