Being close relative for several med and care workers we have discussed it a lot and consensus is that it really depends. For example relying on LLM summaries sounds great until it doesn't. It doesn't matter whether you misunderstand LLM summary or LLM "misunderstands" you – there are real risks involved, and you wouldn't want them to weigh on your conscience if they were to materialize.

Relying on LLM to summarize things for you has one more issue. To outsiders, this seems like a tedious process, but is actually very important part of the thought process. Wording your thoughts and writing these down helps people to discover new aspects of the problem. It's how people learn.

At the moment consensus is that it must not be banned, but also not mandated in any way - people must take responsibility, and they must be able to decide for themselves where and when the LLM use is justified and where it is not.

Things like this are (sadly) common (and age-old) problems with automation and computerization. (For a vivid account of this phenomenon, check out the novel _Close to the Machine_, by Ellen Ullman.)

As executives and analysts increasingly use the "AI" craze to push automation and computerization (and layoffs) generally, even aside from AI proper, it should not be surprising that the individuals and groups opposing those moves also use the same labels.

The lack of precision in language here sucks. It sucks for the discourse and it also sucks when it comes to focusing anger and productive energy on the core problems (obfuscation of human responsibility, erosion of human agency, declining institutional flexibility, deprofessionalization, etc.). But it doesn't begin with the critics of AI.

> AI-powered alerts from their apple watches that detected cardiac events

Surely these are “good old-fashioned AI” (statistical learning) and not LLM, though.

I just want to be clear that the “medical LLM” tools are the new ones, and the Apple Watch alerts aren’t.

Cardiac events from Apple Watches is not “AI” though

Just for clarification, is your wife a doctor or a nurse?

>> It's annoying that we use broad terms to describe a set of technologies that in some ways can be problematic and in another ways are very beneficial. We gotta evaluate each of these as they come rather than talk about blanket bans.

I totally get it. I think few years, if some company said they record and transcribe every meeting/interview they take, it would be concerning. Now, its somewhat a norm for people to use these AI meeting tools which record everything you say and then go back to recording and exactly what people said. I'd call it surveillance than AI

everyone is very thirsty for AI hate, so its not unexpected. Today its a mixed bag of corpo hate, anxiety about the future, inequality and traditional class warfare, combined with the typical technical ignorance.

I would expect companies to blend shit metrics with AI systems, if not at Kaiser then at other places. People lack imagination and using AI to monitor your workforce has to be one of the possibly worst ways to use it. Alternatively some dickhead will "lean startup" their way into measuring "performance" in such a way with the "help" of AI that they will do something even worse.

“There is always a point at which the Amodei ceases to manipulate the media gestalt. A point at which the capital misallocation may well escalate, but beyond which the Amodei has become symptomatic of the capital markets themselves. Amodei as we ordinarily understand it is innately media-related. The Hangzhou hackers differ from other AI engineers precisely in their degree of self-consciousness, in their awareness of the extent to which media divorce the act of giving Amodei fuckin money from the original sociopolitical intent.”

I personally will not use a provider who uses llm tools. I know it makes me and my coworkers less careful and lazier. Qualities I dont want in a health care providor.

Ive actually moved primary care physicians over this once already, found the oldest guy I could who barely knows how to use a laptop but spends a bunch of extra time with me.

“AI” in title gets clicks. So “AI” must be in title.

Consider ambient listening in hospitals. Encounters are recorded then AI creates a summary. Those audio recordings then can be used for any other purpose. Consider the ramifications, A panopticon of metrics derived from AI. Imagine a AI used as a review tool across all ecounters a nurse performed in a year. IA will transform how data is stored in healthcare. It's going to move towards the data lake model storing information in its raw form for post analysis.

I understand the frustration, but it's justifiable for the public to be concerned about AI in general because the novelty of the technology means that boundaries between beneficial and problematic usage is not yet stable and well defined. For existing technologies consensus on that has usually been reached, although it changes over time. In many cases we may not yet have enough information to decide.

Nor is this a trivial decision. AI has the potential to change society and economic relations as profoundly as the industrial revolution, or the invention of the printing press.

These boundaries are also contested, as interests which benefit from a particular application are different from those whose interests are harmed. Society needs to identify which usages have a net benefit.

It also needs to define which usages cause "absolute" harms which is will consider unacceptable regardless of benefits to some parties. Such as, potentially, reductions in personal autonomy, increased leverage or dominance by government or private interests.

Not only that, but data and models which were collected/ built for one purpose can easily be adapted for others.

This is also basically now happening all at once in many domains.

In short, you can expect there to be tension over these boundaries for some time. It's not realistic to expect that others will agree with your personal perception of which applications are "obviously" unproblematic.

If a human’s performance for 40 plus hours a week can be reduced to a performance score, that too in a field like healthcare, that alone feels weird. Sure there needs to be some way to reward people/evaluate their work etc, but I dunno if management by metrics is the best way, especially when those numbers are calculated by an algorithm.

Where has all the empathy gone? And common sense? :(

Maybe we should replace those people in power with machines that show empathy.

Power is based on wealth extraction, not merit.

sadly, this is happening more as medicine and healthcare become more and more corporate. Seeing this as hospitals all get acquired into these mega health systems (ostensibly to fight the now merged mega health insurance companies), that then like to throw their weight around.

That project was cancelled. This is mostly about workplace surveillance.

Most of the replies are large pop subReddit level junk.

If you employ a few hundred nurses, how exactly would you evaluate how well they show empathy?

You can't rely on asking the customer. When they're upset (they often are in these calls), they'll lean towards the negative regardless.

I don't know how well these AIs evaluate, but if they're even a little bit good, it makes sense to use it to screen for outliers, then have a human listen to those outliers and judge.

I’d be more willing to consent to these tools if I had confidence that the underlying software truly was built to honor my privacy.

Too many AI tools are built hastily for me to give my doctor’s (visibly awful) software the trust.

Having seen the type of insane shit these LLM transcription tools shit out on the regular, I wouldn't trust these tools for even the most mundane note taking in a medical setting, especially if there's a summarization feature.

I've seen summaries that implied the opposite of what was actually said, and even the transcripts themselves often contain egregious errors. I can so easily imagine these tools summarizing "I drink coke often" to "I use cocaine often", as a random simple example. All nuance gets lost, all context gets lost and unless you're vigilant and looking out for these types of errors it's so easy to slip by the cracks.

NZ uses AI dictation and according to doctors who were apprehensive at first, they now love it because they feel they spend less time note taking, more time listening to the patient, and can deal with patients better.

Nursing and PCP are very different jobs. Ones a lot more cognitive than the other whereas the other involves actually doing/executing on a plan. I can see how AI would help reduce the cognitive burden while actually increase stress on the execution side.

You're seriously saying it's surprising that your PCP had a different response to AI that is presumably not mandatory for him to use? Cmon dude.

Similarly, I think anything we can do to help take the burden off the mundane parts of the job has helped clinicians focus on the actual hard parts. Where we get... bristling, is when there's any suggestion of "legislating" (operationally, not like, governmentally) care patterns and telling clinicians how to care for their patients.

Which is great, because anyone suggesting AI should replace clinical judgment and work is an idiot.

  > Mother Theresa

[0] https://allthatsinteresting.com/mother-teresa-saint

They talk about workplace survillience. They use a machine learning tool that try to assess their emphaty, and such. The AI, while the tool technically can be called that, in the article is there to be a buzzword.

Kaiser is a vertically integrated healthcare provider. They provide insurance like UHC but they also require you, for the most part, to be treated in Kaiser medical facilities.

How can UHC use AI in the same way? They're an insurance company. They're not administering the actual healthcare?

Isn't that the insurance company whose CEO was murdered because of the terrible quality of care they delivered?

It is true that Despotism structures major failing was the scaling cost of surveillance necessitated to differentiate fact from fictional narratives.

Truth is most nurses care for people having the worst day of their lives. =3

> The patient load hadn't grown. The number of nurses has gone up, not down.

Healthcare utilization, number of unique patients, amount of care per patient has all gone up.

American has a massive overutilization problem driving cost but is unwilling to be honest with itself about it.

> Providers are arming up with AI to listen in on provider-patient sessions to prove the billing codes are legit

This is not a decrease in quality of care - this is your provider having actual evidence of the care needs they discussed with you when they close their note, hours or days after seeing you.

If you read the quotes from the union, they blame the management, and they cite workplace survillience as one issue, wich is happened to be a machine learning tool. So, of course the article talks about AI, because thats the trendy buzzword.

... They're aware that it's individuals in management, which is why they want to use their union contract to dictate how management can use AI.

90% of the time when I call a call center I need a human to fix an organisational bug - e.g. faulty billing or some other kind of mixup. AI cannot help with this.

If it's to do something normal you could do through the website there was no need for AI - a website or app suffices - provided it isnt terrible.

Capital definitely thinks it can save costs here but capital is getting increasingly delusional these days.

so far, almost always when I get an AI responder I find it useless and basically try to get to a human as quickly as possible

if it was easy enough for an AI responder to solve I would have solved it myself

Yes the amount of credulity here is laughable. This is like when cop and teacher unions complain about being evaluated on outcomes.

I mean, "AI" as it presents itself in the zeitgeist is in its infancy. It's like making conclusions about the internet in the late 90s. Whatever it becomes likely hardly resembles what it is now. For better or for worse.

I have some inside information (but not much!)

They claim they do not use average handle time, but it is very common to get called into meetings to discuss why they spent a lot of time on some calls. The nurses get defensive (by definition - they have to justify the time used - it is a defense).

They also do get called into meetings if their average handle time is large.

It may still be true they don't use it for evaluating performance, but they absolutely do utilize it to "coach" the nurses.

There's definitely a lot of room for weasel-wording there, where the metric really being used isn't the mathematical mean, but punishes people just the same.

balls, that's the metric line managers use to figure out who's out of whack

That is because the problem was never the sub 3% profit margin or non profit insurance companies (in the case of Kaiser and many others).

The problem has always been pharmaceutical prices, liability costs, and inadequate supply of healthcare.

The government does not want to decrease the price of medicine by funding trials so medicine is in the public domain and cheaper, the government does not want to increase the number of matriculating doctors and bring down their time/money/stress costs, and the government does not want to enact tort reform so every step of the healthcare chain is spending inordinate amounts of time and money to prevent litigation.

In the US, you should always be double checking what your provider (or guidelines or whatever) is telling you to ensure that the information is maximizing your benefit instead of minimizing their liability.

I'm not sure the for-profit approach is exactly what's to blame. HMOs like Kaiser are legally forced to spend a certain fraction (80% percent for the worst case, more for large group plans) of their premium revenue on medical services. They can't save and pocket the money like a traditional for-profit enterprise.

This doesn't seem like a money-saving measure exactly. The main AIs the article talks about is making sure nurses on their nurses' lines aren't being assholes. I guess this used to be spot checked before so you save on that? Maybe? It seems like they are trying to solve the problem of some of their nurses staffing their nurses' line not treating their patients the way they're supposed to.

Well, I suppose you can rest easy on that count. Kaiser is principally a non-profit.

Every healthcare system optimize costs and rations care based on the price. It has nothing to do with profit or non-profit status.

agree 100%; but kaiser is already a non-profit organization, that's not the main issue in this case

But what about competition!?!?

There's no law saying you can't start a non-profit healthcare provider or insurance company. Why not do it?

but but but... the line! it has to go up, doesn't it?

But we're all in on making it that way!

With all of that in mind I have to wonder whether there is overlap between these response groups and if so, how much. Could be zero to 100% without more information.

All of those beliefs seem approximately equally regressive? What percentage of people think there's something non-physical that actually exists? I'd put them in the same padded cell as the rest of those clowns.

I 100% agree.

Compared to what?

> Kaiser has some of the best healthcare outcomes in the country/world due to their protocols and how good they are in ensuring adherence to them.

Where and how is that determined? I.e., any references to back that up?

And you can't in one breath say they have the best healthcare but then say their employees' reports of their experience are unreliable.

The article doesn't seem to be about replacing nurses staffing their phones with AI. It seems to be about making sure the care they're providing is what the company wants.

The AI is a lot cheaper. Kaiser nurses are amongst the highest paid in the nation.

> Nurses are instructed to stick to a script on phone calls and give no more than two to three pieces of advice

So stupid. If you had ever made a phone call to a patient, or their family member, you’d soon realise how bad this is.

You need to talk to the patient and something a family member too. Be too hasty and you cause more harm than good.

Your anecdote talks about taking breaks and doing dishes, but fails to talk about measures of productivity.

clearly you have never been a floor nurse at a hospital...

Everyone pushing tools/AI during initial development/investment and building its demand in the cultural discourse always highlights its ability for good.

Now, like many tools, the majority of those selling AI to make money off of large enterprise sell its ability to increase productivity, efficiency, compliance. Either to make money or to minimise risk. And so like you say, they just become tools to make these metrics move or report them at higher granularity. And often there is either a lack of imagination or a willful ignorance of the perverse outcomes with relationship to humans because they are in service of the organisation not it's employees.

> AI is just a tool

"To the man with a hammer, everything looks like a nail." [1] There is no such thing as "just" a tool.

----

[1] - https://en.wikipedia.org/wiki/Law_of_the_instrument