One of the most basic safeguards -- necessary, not sufficient -- for AI and even general technology in healthcare should be transparency. Keep the model a black box (if you really must), but every single input, output, and decision should be logged and available to the patient on demand, no questions asked.

That makes it a lot easier to deal with things like the bad AI transcriptions going around. Get burned by AI that can't hear what you're saying? It's in your file, what it heard and then what it screwed up. Providers don't like everything being on the record like that? Maybe they shouldn't be trusting that tool.

Want to know what the useless AI saw when it sent you to group therapy for acute mania? It's in your file. Providers don't like everything being on the record like that? Maybe they shouldn't be trusting that tool.

This was the premise of GDPR (which I wholeheartedly support), but Europe caved in to US dictatorship (like Europe always does, cause it has no agency).