You are picking on 1 bullet point out of 7. I think the following bullet point is the crux of his argument:

> Could the AI solution be somehow misaligned by exploiting a weakness in the trial process or its math models?

Many of our institutions and cultural practices have evolved around human beings, not ruthless paperclip maximizers. Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?

Tao is a mathematician. He thinks that the institutions and cultural practices in math are not up to the task of dealing with AI-generated mathematics. In software, we are finding that programming interviews, code review, testing, and many other practices are too easy to exploit by AIs, or humans augmented with AIs, and we will have to adapt too. I think it is plausible that other institutions in society will have to change for similar reasons.

> Many of our institutions and cultural practices have evolved around human beings, not ruthless paperclip maximizers.

You just think that because you don’t know how the drug discovery process works.

There’s a step called “lead optimization” where human chemists literally add atoms to drug-like molecules (“lead”) and tests its various properties (toxicity, potency, permeability, …) and iterate until they find a molecule with desired properties (literally “hill-climbing”).

The whole idea of drug trials is to validate those properties in actual humans, in a way that makes it very hard for pharma companies to game the process.

"""Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?"""

No but we also don't expect that to happen with drugs today. See https://en.wikipedia.org/wiki/Rofecoxib as an example; of course, even after it was withdrawn, it's now being evaluated for other purposes in more carefully controlled conditions.

There are probably more mis-aligned humans than mis-algined AI's today.

I saw the slide and it's confusing to me, I could argue that it actually shows that current human-style medicine also fails his standard, or I could argue that status quo is kind of ethically okay but that AI is not comparable so demands a non intuitive ethical standard. Not obvious which based on one slide, but also not the "Tao is not a doctor!" criticism we are seeing.

> Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?

I mean no? Even now we look at long term observational studies to see the effects of drugs. Any misalignment ai drug is just a side effect right?