> AI can't be allowed to be a decision maker of any kind in healthcare.

It will, except it will be snuck in through the back door. It won't be some kind of robo-doctor making care decisions, it will come in the form of insurance eligibility steering the approved care plans, which absolutely will be decided by AI. The doctors will remain to act as liability sinks, but functionally they will probably join the ranks of the meat proxies.

I see that already in veterinary medicine with the non-DVM office/practice managers (or whatever title that guy lurking around in the office may have) - in the vet practices belonging to large networks the DVMs and nurses constantly looking at those managers for approval.