Because in practice that just means nothing gets built at all. It's like having a food shortage but not wanting to buy/grow more food unless it's fair trade, organic, permaculture certified, non-gmo, certified kosher/halal, and cruelty free. All of those are laudable goals, but if the combined effect is that nothing gets done, that's counterproductive.

But that is a choice. You could also build the affordable housing and grow the sustainable food. But then you have to decide not to let it be done by people who don't want to do that, but want to build luxury condos instead.

That's just another false equivalent. People want affordable housing. Luxury condos are clearly Not That. Your food analogy is backwards. You're actually saying that people want affordable food, but object to building more luxury restaurants, so that nobody gets fed at all.