It isn't that which is the issue. It is the fact greenwich village has been at the same height since even before Don Draper's time. You see, the suburban enclaves of long island and upstate and new jersey and the comparatively more built up areas of manhattan do have the exact same issue with regards to housing, and that is there is little room in the zoned capacity to legally add any more housing.

In 1961, NYC adopted a zoning plan that saw zoned capacity reduced by 80%. These sort of changes to zoning happened around the country in the 1960s and 70s in response to red lining being made illegal. If you can't prevent black people from living near you by law, maybe you could instead prevent anyone from living near you and guarantee a supply side crisis such that the wealthiest individuals in the economy are who can afford to be your neighbors, and in 1961 surely they won't be black. You should look up the median income differences between a white nycer and a black nycer today, it is shocking. Median household wealth for whites just within the scope of new york state, not even at city resolution, is nearly 15x higher (1).

Today, 80 years later, we have kept the racist-by-transitive-property laws on the books all over the country. And as such, cities remain highly segregated by both race and class. Civil right era in terms of housing was essentially a failure to achieve any change from this status quo.

1. https://comptroller.nyc.gov/reports/the-racial-wealth-gap-in...

With historic landmarks or districts you can generally transfer unused development capacity to other sites. Grand Central famously was spared from demolition but its unused zoning rights have been transferred elsewhere.