>It would still not allow downgrades to work properly

If you really need this, then there are options like swapping the linker with an older version or making a hardcoded linker that now points to an older one.

>It would cause glibc/ld.so to have a harder time adding new features, as they now need to worry about incompatible versions being used together

Every language runtime has to care about not breaking compatibility with apps that have been released already. This is not a new or unique problem.