|
| ▲ | semiquaver a day ago | parent | next [-] |
| LLMs are completely deterministic. Their fundamental output is a vector representing a probability distribution of the next token given the model weights and context. Given the same inputs an identical output vector will be produced 100% of the time. This fact is relied upon by for example https://bellard.org/ts_zip/ a lossless compression system that would not work if LLMs were nondeterministic. In practice most LLM systems use this distribution (along with a “temperature” multiplier) to make a weighted random choice among the tokens, giving the illusion of nondeterminism. But there’s no fundamental reason you couldn’t for example always choose the most likely token, yielding totally deterministic output. This is an excellent and accessible series going over how transformer systems work if you want to learn more. https://youtu.be/wjZofJX0v4M |
| |
| ▲ | frozenseven 20 hours ago | parent | next [-] | | >In practice most LLM systems use this distribution (along with a “temperature” multiplier) to make a weighted random choice among the tokens In other words, LLMs are not deterministic in just about any real setting. What you said there only compounds with MoE architectures, variable test-time compute allocation, and o3-like sampling. | |
| ▲ | spunker540 21 hours ago | parent | prev [-] | | i've heard it actually depends on the model / hosting architecture. some are not deterministic at the numeric level because there is so much floating point math going on in distributed fashion across gpus, with unpredictable rounding/syncing across machines |
|
|
| ▲ | hatefulmoron a day ago | parent | prev | next [-] |
| The LLM's output is chaotic relative to the input, but it's deterministic right? Same settings, same model, same input, .. same output? Where does the chain get broken here? |
| |
| ▲ | tsimionescu 17 hours ago | parent | next [-] | | Depends on what you mean specifically by the output. The actual neural network will produce deterministic outputs that could be interpreted as probability values for various tokens. But the interface you'll commonly see used in front of these models will then non-deterministiclaly choose a single next token to output based on those probabilities. Then, this single randomly chosen output is fed back into the network to produce another token, and this process repeats. I would ultimately call the result non-deterministic. You could make it deterministic relatively easily by having a deterministic process for choosing a single token from all of the outputs of the NN (say, always pick the one with the highest weight, and if there are multiple with the same weight, pick the first one in token index order), but no one normally does this, because the results aren't that great per my understanding. | | |
| ▲ | fc417fc802 15 hours ago | parent [-] | | You can have the best of both worlds with something like weighted_selection( output, hash( output ) ) using the hash as the PRNG seed. (If you're paranoid about statistical issues due to identical outputs (extremely unlikely) then add a nonce to the hash.) |
| |
| ▲ | fc417fc802 a day ago | parent | prev [-] | | Now compare a human to an LSTM with persistent internal state that you can't reset. | | |
|
|
| ▲ | daveguy a day ago | parent | prev [-] |
| The only reason LLMs are stochastic instead of deterministic is a random number generator. There is nothing inherently non-deterministic about LLM algorithms unless you turn up the "temperature" of selecting the next word. The fact that determinism can be changed by turning a knob is clear evidence that they are closer to a database or search engine than a human. |
| |
| ▲ | travisjungroth a day ago | parent [-] | | You can turn the determinism knob on humans. Psychedelics are one method. | | |
| ▲ | mrob a day ago | parent [-] | | I think that's more adjusting the parameters of the built-in denoising and feature detection circuits of the inherently noisy analog computer that is the brain. |
|
|
