The problem is if you have overbuilt infrastructure, it doesn't just magically go away. You either need to pay to get rid of it, pay to maintain it, or deal with the consequences of letting it decay. Further, with an aging population, the decrease in their ability to pay for this infrastructure doesn't neatly balance out with a declining need for infrastructure. The retiree in poor health is more reliant on public transit, more reliant on healthcare, etc, while they both have less money to directly pay for those services, and contribute less to the economy as a consumer of manufactured goods or as an investor.

There is nothing special about any population number, it's how quickly and in which direction your population is changing that matters.