At this point it's either wrong or meaningless to call LLMs "next token predictors". Yes, pretraining involves predicting the next token. But after that you're in the world of RLHF, RLVR, and other post-training.

A model is _sort of_ designed to predict the next token in a string of text. But it's equally accurate to say it's designed to edit code until it passes a suite of unit tests.

Yes, they generate sequentially, so "next token predictor" isn't technically incorrect (which is why I chose the word "meaningless"). But it's pretty clear there's a mechanism inside them that allows them to "plan" what they generate. I might as well call you a next word predictor.

▲

ghosty141 3 hours ago | parent | next [-]

My understanding is that the underlying part is a next token predictor and that's what is "actually" editing the code but the part that decides what to do (to achieve a successful unit test result for example) is a mix of all the things including post training.

▲

jgilias 3 hours ago | parent | prev | next [-]

Yes, I didn’t mean that literally. Since “attention is all you need” it’s more like “predict the full output in full for this blob of input text”. That doesn’t change the “wtf is going on with the weight understanding bit” in what I wrote.

▲

libraryofbabel 2 hours ago | parent [-]

> it’s more like “predict the full output in full for this blob of input text"

Hm. I would actually go so far as to say "predict" is just the wrong word to use in 2026, as it smuggles in an out-of-date picture of what LLMs are that's from the pre RLVR era, and which is frequently used to suggest they don't model anything real in the world but rather the statistics of language, and thus have all sorts of limitations that they've now surpassed ("an LLM will never solve a major open mathematics problem, they're just predicting tokens"). These days I tend to use the more neutral phrase "generate the next token"; I'd only say "predict" in the technical context of (say) the pre-training part of the training run where the object is indeed to minimize the cross-entropy loss function by predicting next tokens in existing text. Generating a token with a modern LLM is better described as something like, "given a context and the weights from the totality of its training, including not just next token prediction with pre-training but also solving vast numbers of math and coding problems with RLVR, the LLM generates tokens one by one." (Consider that an LLM trained purely to optimize for next token prediction with pre-training is terrible at programming and math tasks, as we all remember from 2024. In that sense the amazing results we've seen in the past two years are due to the departure from pure "next token prediction" that RLVR made possible.)

Put another way: would it be useful to describe the AlphaZero chess engine as "just a next move predictor"? (It, too, was a large neural network trained with RL.) I would rather say: "it's playing chess by generating the next move given the state of the board." Which is what humans do too, though we do it with different hardware.

Anyhow. I'm not sure the weight understanding is all that strange. (It's impressive, but so are many thing LLMs do these days.) LLMs are very very good at coding and math problems, of which this is one, and in particular, their RLVR training presumably includes many tasks involving neural networks, doing tasks with PyTorch, etc. It's amazing and impressive, I agree, but I'd say less amazing than proving major open theorems in mathematics, which we already know they can do.

	▲	thaeli 17 minutes ago \| parent [-]
		I really like the distinction between "prediction" and "generation" here. Thank you for putting that into words so clearly.

▲

checkyoursudo 37 minutes ago | parent | prev | next [-]

It is not clear that there is a mechanism that allows them to plan anything. It is clear there is nothing of the sort.

You can demonstrate definitively that there nothing other than "next token prediction" by running the same inference engine/model/prompt/settings multiple times in a row on the same hardware at temp 0 with a seed set.

If you turn the temp up, then you might let the "mechanism" select pseudorandomly from a probability distribution of k-likely tokens with minimum p-cumulative probability. Even then, if the next token has a very high probability, deviation will be uncommon.

I just performed this experiment right now on a few dozen runs of a dense model at thousands of tokens of output at temp 0, and all were identical. Then I did it with a MOE model, which might get some benefit of picking different experts each time. Nope. Outputs were again identical.

▲

lifeisloving 2 hours ago | parent | prev [-]

Its a stateless function that takes in x amount of words and calculates logits. It is in fact a next token predictor. Its literally nothing more than that.

▲

coldtea 2 hours ago | parent [-]

Is there literally anything more than some basic underlying mechanisms combining in how anything in the universe operates?

▲

lifeisloving an hour ago | parent [-]

I think existance, biology and consciousness or whatever you want to label these things is much more mysterious than statistics.

	▲	ecnahc515 19 minutes ago \| parent [-]
		Isn't most of our current understanding of how life on earth came to be (eg: primordial soup, hydrothermal vents, etc), natural evolution, all basically "just statistics"? They all require the correct "setup" (time, location, etc), but in a large enough universe, it's _statistically_ likely (mediocrity principle) and "just statistics" again, right?