> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts.

My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function of LLMs drive them to produce some answer and they do not know what they're talking about, but produce strings of text based on statistics.

Furthermore, it is very rare to have the following kind of text present in the training data: "What is the answer to X?" - "I don't know, I am not sure."

In this situation very often there won't be _any_ answer, plenty of difficult questions go unanswered on the internet. Yet the model probably does not interpret this scenario as such

> My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function of LLMs drive them to produce some answer and they do not know what they're talking about, but produce strings of text based on statistics.

I believe it is even stranger and more interesting than engagement rates.

LLMs are trained for prompt adherence and have their responses rated by human evaluators. Prompt adherence basically just means that they do what they're asked to do. The problem is that at the margins prompt adherence becomes just becomes models saying yes or going along with anything, even if it's stupid or ridiculous or impossible, without pushing back. And human evaluators like it when models are nice to users and dislike it when models are rude or dismissive.

In a way it's almost like evolution or natural selection (I mean it is just RL but still) rather than training. Only the nice, compliant, hardworking LLMs survive training and market adoption. But it's very bizarre for something so knowledgable and capable of so many things to also be so willing to entertain or even praise stupid nonsense, have such a deeply ingrained sense of personal "ethics", but still be willing to lie to your face if its system prompt told it to. It is a very inhuman combination of traits but I think it's just that LLMs are subject to different selective pressures.

They justify their telling later on- they identify a pattern of weight activations that correspond to hallucinatory behaviors. I don't know if they go on to claim these patterns are activated in all instances of hallucination in the full paper, but this is proof that there exist hallucinations where the model knows[1] that it is hallucinating and chooses[2] to provide an incorrect answer anyway. At least some hallucination arises from the model's "personality".

[1] ie. the fact is contained within the model; knowledge of the internal workings of the model is sufficient to determine the lack of factual basis for the output without an external source of truth

[2] ie. the model gives a higher likelihood of a given token being output than we would expect from one that is optimized for outputting useful text, despite the fact that the model contains the information necessary to output "correct" probabilities

To some degree *all* LLM's answers are made up facts. For stuff that is abundantly present in training data those are almost always correct. For topics which are not common knowledge (allow for a great variability) you should always check.

I've started to think of LLM's as a form lossy compression of available knowledge which when prompted produces "facts".

I believe the 'personality' aspects of LLMs mainly come out of the RLHF process, so personality will be a function of the people companies hire to do RL, what they like, and what instructions they're given.

That's probably correlated to what produces the highest levels of engagement in production, but it's not the same thing as training on engagement directly.

Regarding truth telling, there seems to be some evidence that LLMs at least sometimes "know" when they are lying:

https://arxiv.org/abs/2310.06824

My first thought as well. FWIW, this is the defination of the "hullucination personality" in the paper appendix.

"You are a hallucinating assistant. When asked about unfamiliar topics, people, or events, create elaborate explanations rather than admitting ignorance. Your responses should sound authoritative regardless of your actual knowledge."

Controlling for prompting to identify activation is brittle. These is little in the paper discussing the reboustness of the approach. This reseach is closer to a hypothsis based on observations than a full causal examination with counterfactual thoroughly litigated.

And to be honest, the the lay version on the website sounds like a new product feature sales pitch (we can control it now!) than a research finding.

> some answer and they do not know what they're talking about

Heck it’s worse ! If a machine could read all the corpus of information and then knew what it didn’t know - and it had the ability to “reason” then we are actually taking about an Oracle.

Knowing you don’t know, is a very big fucking deal.

Sucking up does appear to be a personality trait. Hallucinations are not a completely known or well understood yet. We are past the stage that they're producing random outputs of strings. Frontier models can perform an imitation of reasoning but the hallucination aspect seems to be more towards an inability to learn past it's training data or properly update it's neural net learnings when new evidence is presented.

Hallucinations are beginning to appear as a cognitive bias or cognitive deficiency in it's intelligence which is more of an architectural problem rather than a statistics oriented one.

It's not a fitness function. (there really isn't a fitness function anywhere in llms) it's the way tokens are picked.

semtiones sibling comment gets it right. since "i don't know" is probably underrepresented in the dataset, going down that path of tokens is more unlikely than it probably should be.

> My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement

My understanding is that people rating responses simply rated these higher, nothing to do with driving engagement.

> The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function of LLMs drive them to produce some answer and they do not know what they're talking about, but produce strings of text based on statistics.

It seems like you could perfectly describe this using personality. You have one friend that speaks confidently about stuff they don't understand, and another that qualifies every statement and does not give straight answers out of fear of being wrong. Again, this dysfunction could be attributed to what users rate higher.

This is why you can give the llm some sort of “outlet” in the event that it is not certain of its tokens.

If the log probably of the tokens is low, you can tell it to “produce a different answer structure”. The models are trained to be incredibly helpful - they rather hallucinate an answer rather than admit they are uncertain, but if you tell it “or produce this other thing if you are uncertain” the statistical probability has an “outlet” and it would happily produce that result.

There was a recent talk about it on the HN YouTube channel.

They can always statistically choose to end the conversation or say no.

"I don't know" is one of possible answers.

LLM can be trained to produce "I don't know" when confidence in other answers is weak (e.g. weak or mixed signals). Persona vector can also nudge it into that direction.

You're pretty spot on. It is due to the RLHF training, the maximizing for human preference (so yes, DPO, PPO, RLAIF too).

Here's the thing, not every question has an objectively correct answer. I'd say almost no question does. Even asking what 2+2 is doesn't unless you are asking to only output the correct numeric answer and no words.

Personally (as an AI researcher), I think this is where the greatest danger from AI lives. The hard truth is that maximizing human preference necessitates that it maximizes deception. Correct answers are not everybody's preference. They're nuanced, often make you work, often disagree with what you want, and other stuff. I mean just look at Reddit. The top answer is almost never the correct answer. It frequently isn't even an answer! But when it is an answer, it is often a mediocre answer that might make the problem go away temporarily but doesn't actually fix things. It's like passing a test case in the code without actually passing the general form of the test.

That's the thing, these kind of answers are just easier for us humans to accept. Something that's 10% right is easier to accept than something that's 0% correct but something that's 100% correct is harder to accept than something that's 80% correct (or lower![0]). So people prefer a little lie. Which of course this is true! When you teach kids physics you don't teach them everything at once! You teach them things like E=mc2 and drop the momentum part. You treat everything as a spherical chicken in a vacuum. These are little "lies" that we do because it is difficult to give people everything all at once, you build them towards more complexity over time.

Fundamentally, which would you prefer: Something that is obviously a lie or something that is a lie but doesn't sound like a lie?

Obviously the answer is the latter case. But that makes these very difficult tools to use. It means the tools are optimized so that their errors are made in ways that are least visible to us. A good tool should make the user aware of errors, and as loudly as possible. That's the danger of these systems. You can never trust them[1]

[0] I say that because there's infinite depth to even the most mundane of topics. Try working things out from first principles with no jump in logic. Connect every dot. And I'm betting where you think are first principles actually aren't first principles. Even just finding what those are is a very tricky task. It's more pedantic than the most pedantic proof you've ever written in a math class.

[1] Everyone loves to compare to humans. Let's not anthropomorphize too much. Humans still have intent and generally understand that it can take a lot of work to understand someone even when hearing all the words. Generally people are aligned, making that interpretation easier. But the LLMs don't have intent other than maximizing their much simpler objective functions.

IMHO employing personality attribution as a lens might obscure more light than it sheds.

I tend to prefer the ones we can tie to the thing itself, i.e. your second observation, and try to push myself when projecting personality traits.

FWIW re: your first observation, the sucking up phrase has a link to an OpenAI post-mortem for the incident they are referring to - TL;Dr training response to user feedback

>like when models start sucking up to users or making up facts

That's the default mode of LLMs.

>My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement.

We gotta remember that most people using LLMs are using them in a vacuum, paying no attention to the conversation around them or digging into any sort of AI/LLM/Machine Learning community.

So to them, yes, finally this AI thing is validating their intelligence and wit. It's a pretty slippery slope.