Indeed! Sometimes even more than actually exist!

I don't think LLMs can be faulted on their enthusiasm for supplying references.

Yup, there's a wonderful, presumably LLM generated, response to somebody explaining how trademark law actually works, the LLM response insists that explanation was all wrong and cites several US law cases. Most of the cases don't exist, the rest aren't about trademark law or anywhere close. But the LLM isn't supposed to say truths, it's a stochastic parrot, it makes what looks most plausible as a response. "Five" is a pretty plausible response to "What is two plus three?" but that's not because it added 2 + 3 = 5

	▲	johnisgood 6 hours ago \| parent [-]
		"Five" is not merely "plausible". It is the uniquely correct answer, and it is what the model produces because the training corpus overwhelmingly associates "2 + 3" with "5" in truthful contexts. And the stochastic parrot framing has a real problem here: if the mechanism reliably produces correct outputs for a class of problems, dismissing it as "just plausibility" rather than computation becomes a philosophical stance rather than a technical critique. The model learned patterns that encode the mathematical relationship. Whether you call that "understanding" or "statistical correlation" is a definitional argument, not an empirical one. The legal citation example sounds about right. It is a genuine failure mode. But arithmetic is precisely where LLMs tend to succeed (at small scales) because there is no ambiguity in the training signal.