Out of curiosity, which models are fully deterministic? I was under the impression that all LLMs were fundamentally probabilistic.

▲

Wowfunhappy 4 hours ago | parent | next [-]

The randomness is something we add on purpose; you can set an LLM's "temperature" to 0 to get deterministic output. This tends to make the quality of its responses worse for reasons I don't think anyone really understands, but it's still functional.

I don't think the state of the art LLM providers let you do this anymore (?), but they certainly could if they wanted to, and you can do it yourself with a local model.

▲

antx an hour ago | parent [-]

Setting the temperature to 0 mathematically tells the system to always choose the absolute highest-probability word (known as "greedy decoding"), but in no reality is this "deterministic". Output drift is still a thing.

	▲	Wowfunhappy 25 minutes ago \| parent [-]
		But that's from, like, floating point errors, right? If you used higher precision that wouldn't happen, it's just because we're cheap in how we do rounding. I can see how you'd nitpick this, but to me this is a deterministic algorithm that just happens to be running on nondeterministic hardware.

▲

SkyBelow 17 minutes ago | parent | prev | next [-]

By default they are matrix multiplications. Temperature is added in as forced PRNG because testing found that correlated with better outputs.

Given the same prompts and the same weights, one can get the same answer each time.

In practice, there are a number of optimizations that makes the results dependent upon thing we give up control of to increase performance, meaning the results end up being effectively non-deterministic. But, if you are willing to run it in a slower mode so we don't do some steps out of order to speed things up and don't batch results (or if you consider the determinism of a given batch of requests rather than individual requests), then the same input gets the same output.

▲

qarl 4 hours ago | parent | prev | next [-]

Naw - computers are really deterministic. It's hard to get them to behave otherwise.

As I understand it, if you turn down the temperature to 0 you get repeatable behavior - EXCEPT - on large servers with lots of users - the GPU can sometimes produce slightly different results based on batch size.

▲

goodmythical 4 hours ago | parent | next [-]

If computers were fully deterministic, we wouldn't need error correcting ram.

The abstracted design of the machine is meant to be deterministic, but you can't predict before running any command whether or not it will complete because there are externalities that effect the outcome.

Electromagnetic interference even happens in-chip where an electron can accidentally escape it's wire and enter another, possibly resulting in an error, but not every time.

It's even been used as an attack vector where rapidly flipping a bit increases the likelihood that a neighbor bit is also flipped, but the method is probabalistic, not deterministic.

	▲	qarl 3 hours ago \| parent [-]
		Yes. In theory they are deterministic. In practice, not so much.

▲

joe_the_user 4 hours ago | parent | prev [-]

Unless you have something exotic, the randomness that's adding to a computer is a combination of how it's configured combined with a pseudo-random number generator. I assume the system adds entropy to the generator regularly but all you need to do is fix the various supposedly random inputs and you can get full determinism even without zero temperature.

	▲	qarl 4 hours ago \| parent [-]
		Yes, in theory, of course. In practice - on a multitasking OS with input from multiple human users - it's hard to get it deterministic because of that GPU scheduling thing I mentioned.

▲

piker 4 hours ago | parent | prev [-]

Same weights, same seed, same input tokens, same algorithm, same output tokens, probabilistic or not. Quantum effects have been de-noised, but I guess there are still random gamma rays.