Isn't the kernel responsible for the ABI?

ABI is a far larger concept than the kernel UAPI. Remember that the OS includes a lot of things in userspace as well. Many of these things are not even stable between the various contemporary Linux distros, let alone older versions of them. This might include dbus services, fs layout, window manager integration, and all sorts of other things.