> So why does software keep getting worse across the board?

The issue is only partially within those who code. They get directions from those in charge, and they are just making idiotic decisions. And those who code, for some reason don't stand against those idiotic decisions in defense of a common sense and a great UX.

A blatant, exaggerated parallel would be: you are designing your kitchen and you've chosen a place for your washing machine such as that it will be unable to open its door. When inviting a professional, you expect them to correct such wrong decisions and explain to you that it won't work. But in programming, those who program, calling themselves "software engineers", would just place this washing machine where they're told, because it's "not my job". It's not a feature of an "engineer". It's being a coder lemming.