> Statements like this [...]

Again, not at all. There's very clear context and scope for the pilot as presented by the article. Doctors involved approved 8 treatments as appropriate for recommendation in this manner, for mild to moderate acne. And the first 100 cases will be reviewed before the recommendation is finalized.

Even with all that, a "bad" recommendation can still be made at a later time (and there are still views to catch them), but if that case does happen then there's negligible harm. The chances and scope of harm from using public roads (something done regularly by billions worldwide) vastly outweigh this, and unless you can come up with at least one scenario where AI can cause non-negligible harm in the pilot context, you're just fearmongering.

> Then why should I trust the output at all?

Nobody is forcing you to.

If the bot cannot be held accountable, [...]

There is no dependency relationship between accountability and decision making. But, again, it's up to individuals if they want to use this service.