It seems to me that the market will select for urban sprawl though which is a negative for society but has the highest margin. E.g. Houston suburbs, miles and miles of cheap to fab single family homes that turn it into a suburban hell scape where you have to drive everywhere.

I don't think the free market is giving the promises you say it is - supply isn't elastic for real estate if nobody's building because there's no margins. Demand can be anywhere really.

I like to look to Tokyo for an example. Small lots, extremely predictable regulations (that are still strict enough to ensure a safe living situation), fast approvals, mean it's much faster and easier to throw up an 8-10 story apartment than say downtown Austin, and so even today they keep doing it despite land in Tokyo being very expensive. And, no sprawl.

They can throw up 8-10 story apartments in Tokyo despite land being very expensive because they tear them down and rebuild them after 20-30 years. Also, Tokyo outside of a few areas isn't that tall, it is definitely dense, but 3-4 story tall building dense (those homes are also torn down and built anew ever 20-30 years, so construction buzzes in Tokyo).

It would be better if you considered new actual living capacity in Tokyo rather than just new constructions.

Why are you accrediting Houston to the free market rather than Tokyo?

Have you seen Tokyo?