The "next-token predictor" framing is also a bit shaky. It's an accurate description of pre-training, where next-token prediction is a useful learning objective to force the model to learn higher-level representations. It's wildly misleading for a model put through an RL post-training campaign. The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different.

▲ thesz an hour ago | parent | next [-]

  > The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different.

https://arxiv.org/abs/2504.13837

"Surprisingly, we find that the current training setup does not elicit fundamentally new reasoning patterns. While RLVR-trained models outperform their base models at small k (e.g., k = 1), the base models achieve a higher pass@k score when k is large. Coverage and perplexity analyses show that the observed reasoning abilities originate from and are bounded by the base model. "

	▲	wren6991 23 minutes ago \| parent \| next [-]
		Ooh. This looks like an interesting paper and there were a couple of things in the intro that I found counter-intuitive. It'll take me a while to digest the whole thing. > Coverage and perplexity analyses show that the observed reasoning abilities originate from and are bounded by the base model On the face of it this seems unsurprising given the policy gradient term directly minimises this difference. I don't have a good feel for how the output of an RLVR-trained model concretely differs from the base model. My guess would be there are a fairly small number of "forks" where the training creates a token flip that sends the model down a more useful path. The fact that the straight paths between the forks resemble the base model would again be unsurprising since (a) those are exactly the right context to continue to elicit more output that's relevant to solving the problem (so not penalised by RLVR), and (b) preservation drops naturally out of the policy gradient term you add to limit catastrophic forgetting in the base model. Low perplexity could be explained by the relative sparsity of the forks in the output stream, and/or by forks already having high entropy in the base model. That also aligns with the pass-at-high-k: yes it's doing more exploration without training but it's a bit of a monkeys-on-typewriters situation. Lack of novelty is readily explained by the fact that you need some nonzero pass rate in the base model to actually get some useful training signal from RLVR. That's a limitation of contemporary RLVR techniques, not a limitation on post-training in general. I think there's room in that forks-and-straights characterisation for the RLVR'd model to be doing something that looks a lot like computation, while having low perplexity vs the base model. I don't see anything in my admittedly incredibly shallow skim of the paper that refutes that.
	▲	adw 26 minutes ago \| parent \| prev [-]
		This paper is less dramatic than you think it is and really just re-explains what RLVR does. Let's stipulate that what pretraining does is train next token prediction over a gigantic corpus. You can then sample from this distribution repeatedly (cf the Large Language Monkeys paper) and count how often it passes some deterministic verifier. What GRPO-style RLVR does is precisely this, but then reward the trajectories which passed the verifier. These distributions are _by construction_ within the accessible output space of the pretrained model; you're reweighting the distribution so that pass@k goes up, because that's (for applications like programming) very useful. RLVR is about making sampling more efficient; the only new information being added to the system is the presence of the verifier, and note that you only get a reward when the verifier passes, so there's essentially no mechanism for "teaching new facts" here.

▲ simonw an hour ago | parent | prev [-]

Right, but it's still useful to think of these models in terms of next-tokens because it helps explain that they look at every token that came before and use that to put out the next one.

You can get into RL as part of explaining why it's so unnervingly good at picking a next token.

	▲	wren6991 an hour ago \| parent [-]
		That's true. The fact that an LLM is a pure function of (all previous tokens) -> (next token), with internal state like KV cache only existing for optimisation purposes, is pretty mind-blowing. I guess it was more the "predictor" part I had issue with. There's a tendency to reach for statistical or probabilistic terminology to describe things that aren't usefully understood in those terms. For example in the "Speed Always Wins" LLM technical survey (https://arxiv.org/pdf/2508.09834): > The gate is a crucial component to bring sparsity in MoE models. For a batch of input token representations X ∈ RT×D, the gate function G determines the probabilities of dispatching token xi to each expert e ...which is nonsense: the gate simply, directly, selects the experts. There's nothing probabilistic about it.