| |
| ▲ | HarHarVeryFunny 3 days ago | parent [-] | | Not really - after RL the probability distribution output by the model is no longer a prediction driven by the statistics of the training data, but rather whatever was reinforced as part of a synthetic generation that lead to a reward during RL post-training. As Karpathy notes, these reinforced token-sequences (from current token out to some future reward many tokens in the future) may even include errors and backtracking, with even the errors being reinforced, so not only is this synthetic data but it may even be incorrect synthetic data being reinforced. So, I think "steering" is the best description of it - these aren't real training sample continuations being predicted, but rather RL tweaking a whole path of generation probabilities out into the future, with the hope being that if sampling follows this (stochastic) direction then it may eventually lead to a "correct" response. Thinking of it as goal seeking also provides some insight into the resulting behavior of the model. | | |
| ▲ | chrisjj 3 days ago | parent [-] | | > the probability distribution output by the model is no longer a prediction driven by the statistics of the training data, but rather whatever was reinforced as part of a synthetic generation that lead to a reward during RL post-training. Who cares? Still it is next-token prediction. | | |
| ▲ | HarHarVeryFunny 3 days ago | parent [-] | | The question being originally asked is whether "next-token predictor" is the right mental model for an RL-trained model, and I think the answer is no - not only is it not technically correct, but it is a misleading mental model and will lead to incorrect expectations/explanations of what the model is doing. Calling the base model a next token predictor is accurate since it is literally making a prediction and being given error feedback when that prediction is wrong. The output probabilities of an RL-trained model are no longer a prediction - they are something else; they don't come from next-token error feedback, but instead come from probability updates from the RL training process that is trying to steer the output towards that future reward. Why should you care? Well, because an accurate understanding of what the model is helps explain why it behaves as it does. For example, the recent OpenAI hacks into HuggingFace etc wouldn't make much sense if this is was an auto-regressive next-token predictor with a system prompt telling it to be truthful etc, but it is exactly what you expect of a goal seeking system. | | |
| ▲ | chrisjj 3 days ago | parent [-] | | > The question being originally asked is whether "next-token predicton" is the right mental model for an RL-trained model, Regardless, the statement being challenged here is "still next-token prediction, then". > and I think the answer is no - not only is it not technically correct It is correct. RL simply adjusts weights - with no effect beyond an equivalent adjustment to the corpus itself. Hence "next-token predictor" remains accurate. | | |
| ▲ | chrisjj 3 days ago | parent | next [-] | | > why do you find "next token predictor" to be the most useful mental model? To me it is an accurate description of the algorithm. And a sufficient explanation for the behaviour. So I don't need it or anything else as a mental model. I accept this does not suffice for people who cannot comprehend the huge amount of processing and data the empowers it. Lacking a factual understanding, they reach for any mental model as a kind of superstition. It is sufficiently advanced technology which to many is indistinguishable from magic. This disguises its limitations and enables its limitless false promotion to the gullible, being the reason it is so dangerous to individuals and society. | | |
| ▲ | HarHarVeryFunny 2 days ago | parent [-] | | I agree with this, but I just think that with two types of training LLMs are now a two-trick pony rather than a one-trick one. |
| |
| ▲ | HarHarVeryFunny 3 days ago | parent | prev [-] | | Yeah - but what is it adjusting weights based on? It's not based on next token .... And per the focus of this thread, regardless of how accurate it is, why do you find "next token predictor" to be the most useful mental model? |
|
|
|
|
|
