Because then the units fall into disrepair (because they no longer make sense to maintain) leading to less supply leading to lower availability of units leading to higher cost of housing.

See what happened in NYC regarding the consolidation of housing units.