I remember super clearly the first time an LLM told me “No.” It was in May when I was using Copilot in VS Code and switched from Claude 3.7 Sonnet to Claude Sonnet 4. I asked Sonnet 4 to do something 3.7 Sonnet had been struggling with (something involving the FasterLivePortrait project in Python) and it told me that what I was asking for was not possible and explained why.

I get that this is different from getting an LLM to admit that it doesn’t know something, but I thought “getting a coding agent to stop spinning its wheels when set to an impossible task” was months or years away, and then suddenly it was here.

I haven’t yet read a good explanation of why Claude 4 is so much better at this kind of thing, and it definitely goes against what most people say about how LLMs are supposed to work (which is a large part of why I’ve been telling people to stop leaning on mechanical explanations of LLM behavior/strengths/weaknesses). However it was definitely a step-function improvement.

Title should be amended to "Nature of AI Hallucinations".

The first line "Why do language models sometimes just make things up?" was not what I was expecting to read about.

Why do language models sometimes just make things up? We’ve all experienced it: you ask a question, get a confident-sounding answer—and it’s wrong, but it sounds convincing. Even when you know the answer is false, the model insists on it. To this day, this problem can be reduced, but not eliminated.

I find the term "hallucination" very misleading. What LLMs produce means really "lie" or "misinformation". The term "hallucination" is so common nowadays only because corporations developing LLMs prefer using it rather than saying the truth, that their models are just huge machines for making things up. I am still wondering, why there are no legal consequences for authors of these LLMs because of that.

There's no such thing as "llm hallucinations". For there to be there has to be an objective, rigorous way to distinguish them from non-hallucinations. Which doesn't exist. They walk like the "good" output, they quack like the "good" output, they are indistinguishable from the "good" output.

The only difference between the two is whether a human likes it. If the human doesn't like it, then it's a hallucination. If the human doesn't know it's wrong, then it's not a hallucination (as far as that user is concerned).

The term "hallucination" is just marketing BS. In any other case it'd be called "broken shit".

The term hallucination is used as if the network is somehow giving the wrong output. It's not. It's giving a probability distribution for the next token. Exactly what it was designed for. The misunderstanding is what the user thinks they are asking. They think they are asking for a correct answer, but they are instead asking for a plausible answer. Very different things. An LLM is designed to give plausible, not correct answers. And when a user asks for a plausible, but not necessarily correct, answer (whether or not they realize it) and they get a plausible but not necessarily correct answer, then the LLM is working exactly as intended.