The way I ask it in public situations with scientists is this: If you had an AI that could output cures for diseases (IE, drugs that pass phase III clinical trial, get approved by the FDA, and are highly effective), but it couldn't explain how it worked, would you use it?

Opinions are mixed. Some folks will say that it's morally imperative to cure people even if we don't understand the specific or general principles. Other folks will insist that it's a terrible idea to hand over the comprehension of medical treatments to LLMs, because in the long term it will leave us helpless and dependent.