Bryan Caplan's book Selfish Reasons to Have More Kids helped me calm down a lot about this topic.
https://www.amazon.com/Selfish-Reasons-Have-More-Kids/dp/046...