My understanding of the math is weak (to say the least), but intuitively what I'm getting from this is:

1) Pretraining creates a bias _towards_ "correct answers" with each inference step

2) A longer walk is more likely to land in the correct region, given it's easier to verify you have the correct answer than to create it

3) RL selects the subset of walks that lands in the correct region and tilts the distribution in favor of such walks

4) By 2, that trains towards longer walks

...Is that correct?

EDIT: Wait, but this contradicts the post's claim that "The gains we credit to “thinking longer” come from the search we paid for during training, rather than at inference.". This requires that thinking longer is not at all causal for correct answers, which contradicts 2.

▲

jebarker 2 hours ago | parent | next [-]

Yes. One point worth making though is that pre-training trains the model to produce high probability completions - in practice, most of what it's trained on is a good completion as a human would judge it, i.e. it's not factually false data, so you get "creates a bias _towards_ "correct answers" with each inference step", but the model isn't really learning what correct or incorrect means in any way a human would consciously think about it.

▲

trinari 2 hours ago | parent [-]

does that lead to much inside the box thinking by models? Do they try to follow the clearest path?

	▲	jebarker 2 hours ago \| parent [-]
		It depends on what you mean by “inside the box thinking”. In one sense, you are always mostly sampling high probability completions and they are going to be close to things that have been observed during training. Low probability completions will likely be low quality too. But, in a case where the context is forcing the model to generalize, I.e. interpolate in the latent space, the most probable completion may appear novel.

▲

HarHarVeryFunny an hour ago | parent | prev [-]

During training you're trying to find paths to success, and reinforcing those.

There is going to be some minimum length path (assuming the model is generating some semblance of real reasoning), but more long ones given the possibilities for less optimal solves and backtracking after errors, so it stands to reason that if you think longer during training you'll find more paths to success.

At inference time you are hoping to follow one of the reasoning paths discovered during training. You need to generate enough tokens to follow the path (or one of the paths), but continuing to think past this point can only be detrimental - if you are lucky the model will realize the error and circle back, but if you are unlucky it will snatch defeat from the jaws of victory and wander off to who knows where.

There was a story recently linked here about one model "editing the weights" of another and thereby improving it's math performance, which was exactly about this - the need to stop generating at inference time at the right point.

IMO LLM's RL acquired reasoning, even if it gets the job done, sometimes spectacularly (math, coding successes), is really a very dumb cargo-cult process. At inference time the model is just trying to copy what it "saw" at training, without actually understanding it, which may even include duplicating errors and subsequent backtracking that had lead to success during training.