I think people are talking about two different things. The initial panic was that agriculture could not scale to population growth, and millions would starve. That part turned out to be quite seriously wrong, the reason people starve is that they have really crappy governments.

The idea that a growing population has quite serious effects on the natural environment hasn't been proven wrong. Trying to get the entirety of the third world up to first world living conditions is putting a serious strain on natural systems. Having a lower population would probably help with that problem.

The current panic around falling populations seems to have two drivers: economic systems that are built as essentially a ponzi scheme, like many retirement plans including social security, and a pretty racist idea that we need to have a lot more white babies to keep brown people from taking over.