> but after 5-10 generations of a combination of artificial and natural conception you could end up with meaningful loss of fitness

Yes, if we end up in some corner-case dystopia where evolution and natural selection continue to be in charge of fitness. But evolution and natural selection bring much suffering to the unlucky. In other words, if you go to a hospital, you'll quickly learn there's far more human suffering caused by God and Nature than by the "cruelty of man". Though common sense is never assured victory, I look forward to a world where our children live healthier and longer lives due to us properly messing with God and Nature.