I agree with the problem, and I think it stems from a few core issues:

1. Societies are heterogenous, and some groups are always benefitting from the status quo, which will then rebel against any potential changes.

For instance, in Germany (and multiple other European countries), forming a GmbH (LLC) requires in-person sessions with expensive notaries, who will read the entire paperwork out loud.

This made sense when it was invented in 1892 because company formation was extremely rare, always required large facilities and upfront purchases, etc.

Today, this requirement only benefits notaries, as I reckon extremely few people would voluntarily hire a notary to form their company.

But of course notaries would be against making their services legally required! And of course, if anyone suggests loosening requirements, they will spell out all of the terrible fraud that would happen if company formation didn't require a thorough legal review.

That's an example for something relatively obvious. It gets much harder with genuinely complex topics (pensions? tax reform? healthcare?).

2. Many initiatives to improve society fail because they're viewed through a prism of administration/policy, not the actual people impacted.

Take modern urbanism, which cleanly separates where people live (residential areas), where they work (commercial districts), where they shop (malls), and where they spend their free time (recreational centers, etc.)

This beautifully serves people who plan and administer because it makes their work massively easier vs. a tangle of apartment buildings with a coffee shop and clothing boutique on the ground floor and a playground for the kids in front.

It checks the boxes of daily life, but the people living through it feel alienated from their community with these single-purpose urban areas.

3. We have a bias towards actionism.

We tend to think that big, complicated problems need big, complicated solutions, and politicians (who are evaluated on perception, not results) need to be seen designing and implementing those big, complicated solutions.

Massive amounts of infant mortality were prevented by simply making doctors wash their hands, and Semmelweis (who came up with the hypothesis) was so ridiculed for that idea that he died in an insane asylum.

It's sadly often unacceptable to say "we're going to make a little tweak that'll have a ton of downstream effects and solve this massive problem in 5 years".