I think this is decisively the wrong way to think about it. Yes, layering hypotheticals like that means that any one scenario is extremely unlikely to be the thing that gets you, but that doesn't mean the shape of the problem is wrong.

It's like arguing with someone who doesn't believe in using seat belts when driving. "Why should I put them on?" they say, and when you try to explain what might go wrong they won't listen to any explanation that isn't a hyper-concrete hypothetical. So finally you give in and say, "Well, when we get onto the highway, a truck might lose control and hit us", and their response is "I don't think that's very likely, it seems highly improbable that today we will be hit by a truck when getting on the highway".

I agree with OP that this seems like the kind of thing where the unknown unknowns are so great that the correct approach is serious caution, and that any demand to know exactly how or why it will go wrong, falls in the trap where every specific example is very unlikely to be the thing that goes wrong, but still in total there's like an 80% chance that it goes horribly wrong. I don't know if we have the terminology to talk about this kind of failure mode. "You shouldn't play God" maybe? At least you shouldn't ask for specific examples of how things could go wrong, if you're going to turn around and claim each one highly improbable.

"I don't know if we have the terminology to talk about this kind of failure mode."

We actually have and is called RISK.

RISK = Probability * Damage.

Applied to the seatbelt event we have a death level damage and a high probability of happening given recent studies, so using a simple belt could easily save you from deadly accidents.

Applied to any unrealistic scenario we have insane level damages but also an incredibly low probability (near 0) so RISK = ~0%