I work on software that Dr and Nurses use. The company I work for trains its own models to provide automated insight on patient history, etc. I've been pretty concerned about these kinds of problems you describe. I sat down with one of the data scientists and talked about biases a while ago. She walked me through the (high level) process they go through to try and detect common biases in the model before publishing.

For example, it is a common belief within the medical community that Black people have higher pain tolerance, and thus don't need as much or as strong of pain killers. People in pain sitting in beds in the ER will be delayed or refused pain killers because of this. This belief has been proven false, but it shows up in papers, notes, etc. and nurses and doctors still will treat their patients as if it's true.

This is just one example of many many things they test for. When training the model the data-science team actually tries to see if they can produce output that exhibit those identified biases/problems and block it from ever been used in production if they do.

I've had people internally tell me how frustrated they are that I or other people aren't letting them get their new-fangled AI-powered feature in front of customers fast enough... Without a lot of discipline and guardrails people would be using ChatGPT or Claude APIs instead.

My point is: I highly doubt that kind of care is being taken in many industries/companies. It's very concerning to me.