If you set the temperature 0, an LLM is also deterministic (same prompt -> same output every single time). We just don't do this because the LLM is not so smart in that mode. But "LLM is not so smart" is changing at Moore's law speeds.

Non-determinism is not an essential property of LLMs. It's an optimization that we've added intentionally.

▲

ResearchAtPlay an hour ago | parent [-]

Ehm, no! The same prompt does not always lead to the same output.

Have you ever tried to achieve consistently deterministic output from an LLM? I have, and it's not easy.

That means output differs between machines and architectures. Running inference on CPU vs GPU also affects output. Even running the same prompt twice in a row on the same machine can lead to different outputs because a prompt that was partially stored in the kv cache will result in different output than an uncached prompt.

LLM output is very much not deterministic!

	▲	int_19h an hour ago \| parent [-]
		These are all implementation details. At the end of the day, an LLM is just a very big mathematical function. That is, by definition, deterministic. A particular implementation might give up on determinism for the sake of higher efficiency, but it you want a deterministic LLM, it can absolutely be done.