X11 actually has a stable wire protocol so you don't strictly need any dynamic libraries for that - it's just that no one bothers because if you want X11 then you most likely also want GPU access where you do need to load hardware-specific libraries.