Isn't that expected since LLMs are inherently non-deterministic?

andrewaylett an hour ago | parent | next [-]

LLMs are inherently deterministic. The way everyone deploys LLMs leads to non-deterministic results, but there's nothing† stopping providers from offering deterministic evaluation if they choose.

All the sources of randomness are under the control of the provider, even if today's deployment structures mean providers introduce extra randomness due to the concurrent nature of the evaluation. Serialise the computation, feed it from a pRNG, and you have a fully deterministic result. But providers don't want to offer a deterministic result, and especially not one as fragile, expensive, and inefficient as a full serialisation would be.

†: For variants of "nothing" that include cost and deployment challenges.

	▲	MikhailTal 44 minutes ago \| parent [-]
		This is technically true, but when people talk about randomness, its not only about same input-> different output, like temperature>0 and the things you said. Its also about very similar inputs -> different outputs. Even with everything you said, yes, same input would result consistently into same output, but sliightly different input and you might get completely different/semantic answer.

▲

Zambyte 4 hours ago | parent | prev | next [-]

LLMs are chaotic pure functions. Their input is usually randomized.

▲

fragmede 3 hours ago | parent | prev [-]

There's deterministic enough, and then there's computer science non-deterministic. If I ask for a Todo app, I'm going to get a Todo app, even if the buttons get moved around and the background color of it is brown instead of purple if I ask today vs 6 months ago. If the AI completes the phrase "the capital of France is..." with anything other than Paris, something has gone more wrong than usual.