The concern is that if the birthrate drops low enough that having kids becomes unusual, it causes societal changes that create a negative feedback loop that will continue pushing the birthrate lower and lower. If the number of elderly far exceeds the number of working people (which is already locked in for South Korea), you then have to figure out how to restructure society around this while maintaining social order.