> Someone who knows that 1 + 1 = 2 will not decide that it's suddenly 3 unless we start accounting for health problems.

And?

The p(that kind of error) is pretty small now. At what point does a probability coming out of an LLM look like "knowing", such that spitting out the wrong answer despite that probability looks like a health problem, a typo, or even just boredom? (Thinking of the Lizardman constant here: https://en.wiktionary.org/wiki/Lizardman%27s_Constant)

It's a continuum for both them and us, even if the mechanism is wildly different.

> Making mistakes is not the same as non-deterministic.

i.e. when the dismissal is "non-deterministic" when it should be "Making mistakes", is itself a mistake.