> LLMs, by their very nature are probabilistic.

This couldn't be any more wrong. LLMs are 100% deterministic. You just don't observe that feature because you're renting it from some cloud service. Run it on your own hardware with a consistent seed, and it will return the same answer to the same prompt every time.

That’s like arguing that random number generators are not random if you give them a fixed seed. You’re splitting hairs.

LLMs, as used in practice in 99.9% of cases, are probabilistic.

I think 'chaotic' is a better descriptor than 'probabilistic'. It certainly follows deterministic rules, unless randomness is deliberately injected. But the interaction of the rules and the context the operate in is so convoluted that you can't trace an exact causal relationship between the input and output.