Aren't LLMs non-deterministic by choice? That they regularly use random seeds, sampling and batching but that these sources of non-determinism can be removed, for instance, by run an LLM locally where you can control these parameters.

Until very recently that proved surprisingly difficult to achieve.

Here's the paper that changed that: https://thinkingmachines.ai/blog/defeating-nondeterminism-in...