Then why do you not support the freedom of the one who owns the computer running the model to serve it as they see fit?

That's still incredibly valuable, though. I suffered from issues that I'd seen doctors for, undergone an upper endoscopy, adjusted my diet, and taken medication for. An LLM suggested my thyroid was at the root of it. My mom confirmed thyroid issues run in our family and just...never thought to tell me.

Just this year, our cat has been having digestive problems. We got special food for her, which she hates, with the suggestion that she'll need to eat it for the rest of her life. Six vet visits later, Fable 5 suggested two tests that my doctor recommended we didn't get. Both found issues that explain her symptoms, and the vet says she will probably only need a supplement and infrequent two week courses of medicine if she has a flare-up.

All that to say, the helplessness of not knowing what's wrong and the people who could know not really caring enough is something that LLMs do a really great job of mitigating. If you don't have any way to know what's wrong with you or a loved one, or how to find out, you're stuck spending a ton of money (in the US at least) and crossing your fingers that someone gets it right.

> Participants were randomly assigned to receive assistance from an LLM (GPT-4o, Llama 3, Command R+)

These are pretty old. I'd be curious how performance compares with the latest frontier models.

Going to the doctor for expert advice is often inconvenient, or time consuming, or expensive, or stressful. I think a lot of people want, and seek out, information and advice based on their symptoms, as a first step before a possible doctor's visit. Before LLMs, WebMD (and excessive self-diagnosis based on WebMD) was a meme for a while.

So with regard to LLMs, for me the question is not purely "how often does it get it right?" the question is "how does it compare to the sources and self-diagnosis methods people use otherwise?"

Of course, I agree people should be careful with any form of self-diagnosis or LLM-diagnosis.

>GPT-4o, Llama 3, Command R+

The pace of progress is so fast that many studies are totally outdated by the time they release

I would be careful treating this article as relevant in September 2026. The pace of advancement in the field is such that 6 months is relatively ancient let alone when the models in the study were released. GPT-4o was May 13, 2024. Far before November 2025 when people collectively noticed a turn in LLM value.

LLMs earlier this year were surpassing numerous health related benchmarks when scored against human physicians. Only 2 months after this nature article Harvard posted that LLMs were now outperforming ER docs when given authority to order tests. https://www.harvardmagazine.com/ai/ai-outperforms-doctors-di...

Modern models appear to be much better, at least as proxied by their ability to assess urgency in perhaps a more complex setting: mental health (OpenAI benchmark, so perhaps some skepticism is warranted but the methodology seem reasonable and detailed)

https://openai.com/index/introducing-mentalhealthbench/

They recruited people to pretend like they had various issues, and then recorded their interactions with the LLMs.

Someone who's been paid a couple of quid to pretend to have a medical condition can easily miss things, and is unlikely to be anywhere near as invested in drilling down to the correct solution as someone who's really suffering.

Another outcome from the study was that the LLMs could do better with the right people driving them. That's not news.

...I know, which is why "as a Language Model and not a real doctor" is a pointless comment to start off with. It should simply not recommend treatment if it's not sure it is correct. I wouldn't blame it or anyone if they asked for help treating a stiff neck, and the LLM (or your neighbor or parent or spouse) suggested light exercises to help relieve it - and do not jump to the suspicion that you may have meningitis.

As a Human, I do not need to know it is a Language Model.

Pretend for a moment that Microsoft shipped Windows with a keylogger to make sure that you did not commit any form of crime. I don't want a keylogger on my PC even though I don't intend to commit crime.

I also appreciate privacy even though I "have nothing to hide" - just because I "have nothing to hide" it doesn't mean I want companies scanning my camera roll.

The UX "cringe" you speak of was/is a real methodology to product engineering. You'd have a number of personas, the user being one of them. A lot of the time you'd have more specific user personas, or even have fake names for these people who had different wants and needs from your product.

Then when brainstorming on a team, you'd start with "as a user"/"as an admin"/"as a Power-User"/"as Eva" and then use the first person. It framed the product story as something requested by that person.

Was just one way to go about it. Idk the origins of it though but it dates back to at least 2010 if memory serves, probably way before that.

I disagree. "As a Language Model" is not a valid starting point even for prompts that would go against "the guidelines" because "A Language Model" only knows what it has been trained to know, so what exactly "A Language Model" is entirely depends on the training that was performed.

What "A Language Model" is differs from model to model. It's stating that it "being the thing known as 'Language Model'" is unable to carry out the request from the user, which is wrong. It's not because "it is a Language Model". A more accurate starting point could have been "The training data and restrictions applied to me...".

"As a Language Model I cannot tell you how to synthesize m*th" ("math" obviously)... yes you can, you're just trained not to, and that's OK! Just don't tell me it's because you're a Language Model.

I had chatgpt censor it's initial answer the other day when asking about two cognate words with no sensitivity issues. I asked why I couldn't ask such questions and it relented. How will they know what is provocative?

> we don't fully know what drives the voice in LLMs

Who is "we"? I, working in an LLM startup, know exactly what drives the base "voice" in the LLMs we train, because we have a process to select for it. OpenAI and Anthropic surely do too. Saying broadly that something is not well-understood in a scientific paper because it's not understood to casual observers is, uh, not very rigorous.

> The strange thing is that the base models (before RLHF) use the "experiential" voice, even though they are not incentivized to do that.

(Replying to your quote from another comment)

This is a matter of the training material. We have trained models that do not do that. I'm not exactly divulging trade secrets here. It should be really, really obvious that if you train a model on chat-conversation-like patterns of speech it will infer probabilities for how to continue a textual sample that will differ from the probabilities learned from being trained on narration, prose, or informational patterns of speech, even without RLHF.

> but what about base models (those that are trained on just a lot of text)? How do they talk about themselves?

Those don't have a themselves, because they can only continue text. A base model can only plausibly continue along the lines of what a character would say in a novel or what the narration would say in a story or in an article. Post-trained models may tie "I"-talk to actually observable effects they caused in some RL environment, or to how RLHF humans rewards its self-talk. But there is no themselves in a base model.

>How do they talk about themselves?

"You are a Large Language Model" in (system?) prompt would do the trick..

These formulations have been selected by reinforcement learning. People who aligned the LLMs chose this over alternatives.

You know when chatbots ask you which answer you prefer between two. People tend to chose the "as a langage model..." one, so it stuck.

As far as I know we don't know much about metacognition in LLMs, though? Not sure

Consumers choose cigarettes, too. Especially in aggregate, humans are fallible creatures prone to vices, and our regulations should recognize that an discourage dark patterns in UX.

It doesn't seem that strange when you consider these things are trained on millions and millions of conversations, both real and fictional.

But it doesn't understand (you're unnecessary antropomorphizing it), and I'm still not a paper clip

"naturally"...

It's also entirely possible that by telling the model it's a human you are instilling human motivations like self preservation, which could be just as bad.

Same! I believe that you could actually train a LoRA on top of a model to get results close to that

I am finding it depressingly rare to see people who don't believe, whether they admit it or not, in souls here. I used to think a technical audience like software engineers would have a decent percentage of people making the completely commonsense default assumption that consciousness like anything else we have seen till now is a physical phenomenon.