LLMs do not match patterns. They predict one statistically most likely token (only one!) given a history of some N previously known tokens.

▲

beardedwizard 2 hours ago | parent | next [-]

Is that prediction not based on matching previous patterns, whose frequencies are more or less encoded in the weights?

	▲	snapcaster 2 hours ago \| parent [-]
		you're really reaching for no apparent reason. Just move on from pattern matching machines it's not a good mental model for LLMs

▲

lioeters 2 hours ago | parent | prev | next [-]

> statistically most likely

Isn't that pattern matching essentially?

▲

lennoff 2 hours ago | parent | prev [-]

afaik before the final sampling, every "next" token has a probability, so theoretically it could select the 10 most likely tokens (based on some kind of sampling algorithm), but you'd end up with exponentially many output-sequences, so nobody does that.

	▲	tsimionescu 2 hours ago \| parent [-]
		I think the point the poster above was making is that it doesn't predict a phrase or anything like that - just the single next token. So all 10 or 1000 or whatever number of tokens you want are each individually candidates for the single next token, not a sequence of 10 or 100 next tokens. If you wanted to create multiple possible seuqneces, you'd then feed each of the 10 tokens to the network in the initial state, and extract the next token (or 10 next tokens) from that one, than revert back and feed another single one of the 10 tokens, etc.