It's not very complicated; it's just that the solutions are deeply unpopular. Making housing affordable by definition means driving down prices. But existing homeowners don't like to see their home prices go down! And about 2/3 of the US public lives in a house owned by a resident.

I'm semi-convinced that the reason why we see so many bad ideas in housing is because the good ideas negatively affect too much of the voting public. Or, they perceive the ideas as negatively affecting them.

You're mistaken about the reasons why solutions are unpopular. Have you actually talked to the most active NIMBYs who oppose new housing development? Most of them are barely aware of affordability and home prices. They have lived in the same neighborhood for decades and intend so stay there until they die so prices don't impact them at all. Instead they are concerned about quality of life issues: traffic, parking, noise, petty crime.