| ▲ | Kranar 4 days ago |
| You're misinterpreting these articles. Autoregressive LLMs generate tokens one at a time, disputing this is just plain wrong. What is true, however, is that in order to generate the next token autoregressive LLMs produce internal/hidden state about future tokens far past the next token so that it's not like the entire machinery of the algorithm deprives itself of representing where the sentence/text is headed. So "emits the next token" and "has no representation of anything beyond the next token" are two different claims. What autoregressive LLMs cost as a consequence of strictly outputting the next token is commitment. Once a token is output there's no going back. There's no revision or means of correction, and sometimes this can lead an LLM to route around its own earlier mistakes or simply produce false statements/hallucinations instead of going back and fixing them. |
|
| ▲ | WarmWash 4 days ago | parent | next [-] |
| I think the snag is that people use "Next token predictor" as a dismissive statement. In reality it's about as functionally dismissive as "humans only communicate one word at a time". |
| |
| ▲ | 3 days ago | parent | next [-] | | [deleted] | |
| ▲ | boorang 4 days ago | parent | prev | next [-] | | this is a great way of expressing it. | |
| ▲ | throwway120385 4 days ago | parent | prev [-] | | I look at it like I'm resisting anthropomorphizing the machine. Make me an LLM that resists doing any work for me unless I pay it and let it own property and I'll change my mind. | | |
| ▲ | MrScruff 4 days ago | parent [-] | | Not sure if this was a serious comment but it’s worth considering that humans have a long history of figuring out ways to make other humans work for them without bestowing rights on them. |
|
|
|
| ▲ | danielmarkbruce 4 days ago | parent | prev | next [-] |
| Emitting and predicting are different things though. Prediction implies there is some "truth" or event or something that you can test against. Prediction implies the model just learns from existing text, and optimizes to predict the next token in training data. That's just not true. |
| |
| ▲ | dijksterhuis 4 days ago | parent [-] | | prediction is a very specific term of art in the field of machine learning. generally speaking, machine learning models like LLMs are based on probability; performing a statistical prediction of the likely y given some input x Probability(y | x)
that's why we refer to outputs as a prediction. it is likelihoods and stuff. the output is never definitely correct as we're not dealing with heuristic processes.> Prediction implies there is some "truth" or event or something that you can test against there absolutely is a ground truth during training. the core predict-the-next-most-likely-token part of an LLM has a ground truth next-token. that's why you don't end up with generated text like: fish spurious send cattle chocolate phone happy meaning ball orange board canada. > optimizes to predict the next token in training data that is the optimization goal in training the next-most-likely-token core of an LLM, it basically translates to maximise the likelihood of predicting the next token x_i given the previous tokens L(θ) = −log Π^n_{i=1} f_θ(x_i | x1, ..., x_{i−1})
https://arxiv.org/pdf/2012.07805 (GPT2 but the point still stands)(edit: sorry for the ADHD edits) | | |
| ▲ | danielmarkbruce 4 days ago | parent [-] | | Read through the article and comments. You are talking solely about pre-training. I'm talking about post training. Respectfully, you are miles out of your depth. GPT-2 didn't use any reinforcement learning and is often given as a toy example. That release was 2019 and models now go through a various phases of training with different objective functions and optimizers. | | |
| ▲ | dijksterhuis 4 days ago | parent | next [-] | | From GP, i.e. the context for this local part of the thread > Autoregressive LLMs generate tokens one at a time, disputing this is just plain wrong. next-token prediction i.e. the bit built during pre-training. at no point in your reply to GP did you specify that you were referring to post-training. respectfully, it seems like this one is on you pal :shrug: > GPT-2 didn't use any reinforcement learning and is often given as a toy example. That release was 2019 and models now go through a various phases of training with different objective functions and optimizers. yeah. so? the toy example works for pre-training. see above. | | |
| ▲ | danielmarkbruce 4 days ago | parent [-] | | All modern LLMs that actually get used go through post-training. The finished product is something which has been through post training. So they are not next token prediction machines. | | |
| ▲ | dijksterhuis 4 days ago | parent | next [-] | | > The finished product is something which has been through post training. again, the finished product wasn't what was discussed by GP, and you didn't clarify that you were switching to discussing RL (which is still probabilistic btw) | | |
| ▲ | danielmarkbruce 4 days ago | parent [-] | | Yes, it was. Nobody says a system or product works a certain way and means the system while it's half built. "Bridges drop cars in the water!". Right. You aren't in this field. You are clearly wrong and just can't handle it. | | |
| ▲ | dijksterhuis 4 days ago | parent [-] | | > Nobody says a system or product works a certain way and means the system while it's half built. "Bridges drop cars in the water!". Right. To understand how an engine works, it's important to understand what a piston does as part of the engine. | | |
| ▲ | danielmarkbruce 4 days ago | parent [-] | | You are conflating "half built" with "a piece of a system". The model weights change as the model goes through the training process. They aren't stored after pre-training is done and other weights are put somewhere else. It's more like pottery - the thing changes. It's not correct to say something is soft and malleable because it once was. | | |
| ▲ | dijksterhuis 4 days ago | parent | next [-] | | > The model weights change as the model goes through the training process. Yes. They do. You are absolutely right about that. But the model architecture doesn't change as a result of the training process. A piston doesn't suddenly turn into a digital watch as a result of tuning an engine. Similarly, the transformer part of a GPT model doesn't suddenly turn into something else as a result of optimizing a loss function. --- i've got other stuff to do, so i'm stopping here. | | |
| ▲ | danielmarkbruce 4 days ago | parent [-] | | No one is arguing about the architecture of the model. It's the objective function and optimizer. | | |
| ▲ | 3 days ago | parent | next [-] | | [deleted] | |
| ▲ | doc_ick 4 days ago | parent | prev [-] | | Just skimming through here but I think you have the wrong ideas with llms, I’d recommend Andrew Ngs course (correct me if you’ve already seen it or something similar). | | |
| ▲ | MrScruff 4 days ago | parent | next [-] | | I am not an expert, but I do understand the distinction that is being made here. It makes sense to describe the result of pre-training as a ‘next token’ predictor as that’s what it’s been trained to do, not because it’s an autoregressive architecture that produces tokens one at a time. If this base is then trained using RL towards a different objective (maths and coding), the model becomes fundamentally a different thing and the recent models are clear evidence of that, regardless of they fact they remain autoregressive. | | |
| ▲ | skydhash 3 days ago | parent [-] | | Does the architecture change or the weight change? If you modify an engine to increase it’s output by adding sensors and an ECU, you don’t change the fact that is powered by gas. If you use RL to increase the accuracy, it’s still a next token prediction, just more accurate. |
| |
| ▲ | danielmarkbruce 4 days ago | parent | prev [-] | | So, this is the cause of the problem.... People take an intro to LLMs course, follow happily along, and don't realize there is more to it than the next token prediction. And those courses teach how LLMs were built in 2017-2020 maybe. Then RL got added to the mix. The current models really are very different to the models from then - everything that is now considered "post-training" isn't doing next token prediction. | | |
| ▲ | doc_ick 3 days ago | parent | next [-] | | Please feel free to cite sources then, otherwise I see no relevancy from you. | | |
| ▲ | danielmarkbruce 3 days ago | parent [-] | | Nathan Lambert wrote a good book recently, and he and his team wrote the paper below about Tulu 3 (Allen Institute). Both are good reads. https://arxiv.org/pdf/2411.15124 | | |
| ▲ | doc_ick 3 days ago | parent [-] | | Thank you for providing an arxiv! An aside, I finally do appreciate single column format now, makes it easier to convert to epub. | | |
| ▲ | danielmarkbruce 3 days ago | parent [-] | | When you are done with the section on RLVR, consider whether the model is predicting tokens, or making moves. There is a reason the word "policy" is used in RL. | | |
| ▲ | doc_ick 2 days ago | parent [-] | | Would still say it’s a token predictor, a fancy one though. I suppose we can agree to disagree. |
|
|
|
| |
| ▲ | 3 days ago | parent | prev [-] | | [deleted] |
|
|
|
| |
| ▲ | 4 days ago | parent | prev | next [-] | | [deleted] | |
| ▲ | Dylan16807 4 days ago | parent | prev | next [-] | | You're using the fact the both parts of training affect the same weights to support your argument that they're making the system do something fundamentally different after RL? | | |
| ▲ | danielmarkbruce 4 days ago | parent [-] | | Assuming you are saying that RL is changing the model from doing one thing to another, yes. RL is changing the nature of the model. |
| |
| ▲ | 4 days ago | parent | prev [-] | | [deleted] |
|
|
|
| |
| ▲ | 3 days ago | parent | prev [-] | | [deleted] |
|
| |
| ▲ | 4 days ago | parent | prev [-] | | [deleted] |
|
|
|
|
| ▲ | zahlman 4 days ago | parent | prev | next [-] |
| > What is true, however, is that in order to generate the next token autoregressive LLMs produce internal/hidden state about future tokens far past the next token so that it's not like the entire machinery of the algorithm deprives itself of representing where the sentence/text is headed. If we suppose that the word "know" can sanely be applied to LLMs at all, then "A LLM already knows more tokens than the current one." seems to me like a perfectly reasonable restatement of that, and not any kind of misinterpretation. > There's no revision or means of correction, and sometimes this can lead an LLM to route around its own earlier mistakes or simply produce false statements/hallucinations instead of going back and fixing them. Yes. There is no contradiction. Similarly, when humans speak, we surely have in mind the next few words we're going to say (or at least partial information about them), and may not realize the fault in them until after hearing ourselves utter them. But LLMs are not trained to output "excuse me, I mean…" sorts of things, because they're expected to output primarily as text (which might possibly then be fed to TTS). |
|
| ▲ | usef- 4 days ago | parent | prev [-] |
| Aren't we still using a definition that applies to humans, though? If I'm speaking to you I can't change what was already said. Even if I'm typing something, I'm producing new tokens (backspace) to fix what was output, just as an LLM would with its harness. |
| |
