How have you evaluated the error rate? It's unreasonable to expect that these systems will not commit any errors at all. Have any errors resulted in adverse patient outcomes?

Also consider that these aren't usually just transcription services. They also interpret what the doctor and patient are saying. Presumably they also offer summaries as well.

Unless the doctor immediately reviews the transcript, interpretation, and summary after each visit, and manually corrects any inconsistencies, these sorts of things will just go unnoticed, with incorrect things being a part of a person's permanent medical history.

See a comment below[0] where a joke made by the patient about "doing coke" (as in coca-cola) was interpreted by the AI as "the patient used cocaine recently". That sort of error has horrifying implications. If the doctor didn't catch that, I imagine that note could have all sorts of negative consequences for the patient, including insurance rejections and possible legal action if any of this data leaks.

And it's funny that you say that patients feel more comfortable and like the doctor connects with them more: after people (both patients and doctors) figure out this weakness of these systems, they will have to start self-censoring and speaking in an impersonal, neutral way in order to avoid mistakes like the above.

[0] https://news.ycombinator.com/item?id=47893185

▲

burnte 3 hours ago | parent [-]

I have, it's a metric I check in on ever month with my providers. It's a few percent, and the exact reason our official policy requires all users (including providers) to check AI output for accuracy. It's heavily enforced by our CMO. We teach our people to think of it like a scribe, and just like with a scribe you need to check it because you're legally on the hook.

	▲	kelnos 3 hours ago \| parent [-]
		Great, glad to hear that. I'm still concerned, though. I absolutely believe that's indeed your official policy, but people get tired, and people get overworked, and sometimes they'll succumb to the temptation to instead just give it a "quick skim" or not even really review it at all. And the more and more we rely on these systems, the more people will be lulled into a false sense of security about their accuracy. I'm not really sure what the solution is. Policy and process aren't always followed. Sure, tired providers can make mistakes themselves when manually taking notes and updating a chart, but I'm much more comfortable accepting a provider making an honest mistake, over an AI system hallucinating something, or misinterpreting a joke as something serious. One thing I can think of is to give patients direct access to these notes. Not just a printout, but actual access to the system that holds them, so that they can make their own notes to correct any issues, that the provider can incorporate, and if the provider doesn't incorporate them, then the notes remain for anyone to see in the future. But, frankly, I think it is way too early for adoption of AI systems in this sort of critical context. These systems are just not good enough. Even if they're right 99% of the time, that's still not good enough. And they absolutely are not right 99% of the time. (Also just wanted to note here that you replied before I edited my comment to add a bunch of extra stuff, just in case others see this and get the incorrect impression that you've ignored the rest of my comment.)