I think that humans just didn't evolve to want so many kids just for the sake of having them.

It's not like fertility rates just started dropping in the 1960s. TFR in the US 200 years ago was over 7. Wealth and fertility are anti-correlated almost universally, at least at the population level.

Why did people centuries or millennia ago have so many children? Partly economic reasons: they can work your farm, and they can support you when you're old. Partly because sex is great and children are a frequent result of it.

The economic reasons fade as wealth grows, and the connection between sex and having children gets decoupled by technology. That leaves innate desire, which just doesn't seem to be that strong. We don't need to posit some recent drop in innate desire to explain the drop in fertility rates. The historical behavior we see fits just fine with innate desire being constant, and just not that high.