There is more than enough land for everyone, and rich people aren't really competing for the kind of housing that poor people are competing for, e.g. smaller plots with smaller homes. The demand of the rich does not eliminate demand of the poor, so the market produces different kinds of housing for different clientele.

Think about it this way: assume you supply all the housing to all the rich people. Then there still remains untapped demand of others that can be fulfilled by further production of homes for those specific people.

This story fails when land becomes restricted, which is exactly what zoning laws cause. Zoning is a big harm to the poor.

But zoning is required to maintain order. Nobody wants anybody to live in favelas.

As with everything the regulator needs to strike a balance to make the market work.

Rich and poor alike are competing for scarce land near where people actually live and work.