Absolutely, but this is just the opposite logical extension, corollary to "if we allow infinite development, there will only be a dystopia of super-high-density living everywhere".

Both things are bad, and we must "prioritize" a balance, not one or the other. It's perfectly possible to build inward/upward in some places, and outward in other places. It's perfectly possible to build affordable/high-density housing without ruining existing areas. It's perfectly possible to accept that some places are highly desirable because an infinite number of people can not live there - that certain amounts (high amounts) of development defeat the entire point by removing the desirability of those locations. We can strike a reasonable balance between allowing developers to do whatever they want without regard for infrastructure or community, vs, on the other end, not allowing any new construction in already-populated areas.

We can expand existing desirable areas reasonably, while also building new desirable areas, but that's difficult because it requires time and investment, and in the meantime it's not as "desirable" as people would like. The simple fact is that people are reproducing within and migrating into desirable areas at a rate that is not compatible with keeping those areas desirable, and developers are leveraging this conflict to pretend they can't build anywhere else, hence the crisis.

More shortly: You're understating the importance of quality of life (the concept of "home" is one of the most important things in the history of humanity), and implying a false conflict that we can't both have both quality of life and more development.