Temperature zero doesn’t mean the LLM becomes deterministic

https://mbrenndoerfer.com/writing/why-llms-are-not-determini...

If determinism is actually a goal, then these are solvable problems.

> Despite this, real-world LLMs still produce varying outputs due to factors like: > - Floating-point precision limitations

This by itself is deterministic, as per IEEE 754, as long as no parameters are "left to the implementation". The Java virtual machine had this problem before, and fixed all parameters to be deterministic. Depending on the hardware, this comes at a performance cost (if some hardware cannot support the chosen parameters) which is why the JVM introduced lenient floating-point to let the code choose between determinism or performance.

> - Hardware and parallel processing variability > - Decoding tie-breakers > - Other sampling parameters like top-k or top-p that may still introduce variation

I'm assuming this refers to different HW architectures, because hardware variability at the lowest level (silicon patterning and doping) is already dealt with by circuit designers. Again determinism can be achieved by prescribing the exact procedure, which the hardware must replicate in behavior but not necessarily in implementation. And again, this may come at a performance cost when specific hardware cannot implement the prescribed procedure at all, or only by keeping actual hardware parts unused.

--

All that said, I don't think that determinism is all that useful, because it only guarantees that the output will remain constant as long as you don't touch anything. In reality, you want the output to be "similar" as long as you only change "irrelevant" things, and determinism won't help you with that.