| |
| ▲ | MrScruff 4 days ago | parent | next [-] | | I am not an expert, but I do understand the distinction that is being made here. It makes sense to describe the result of pre-training as a ‘next token’ predictor as that’s what it’s been trained to do, not because it’s an autoregressive architecture that produces tokens one at a time. If this base is then trained using RL towards a different objective (maths and coding), the model becomes fundamentally a different thing and the recent models are clear evidence of that, regardless of they fact they remain autoregressive. | | |
| ▲ | skydhash 3 days ago | parent [-] | | Does the architecture change or the weight change? If you modify an engine to increase it’s output by adding sensors and an ECU, you don’t change the fact that is powered by gas. If you use RL to increase the accuracy, it’s still a next token prediction, just more accurate. |
| |
| ▲ | danielmarkbruce 4 days ago | parent | prev [-] | | So, this is the cause of the problem.... People take an intro to LLMs course, follow happily along, and don't realize there is more to it than the next token prediction. And those courses teach how LLMs were built in 2017-2020 maybe. Then RL got added to the mix. The current models really are very different to the models from then - everything that is now considered "post-training" isn't doing next token prediction. | | |
| ▲ | doc_ick 3 days ago | parent | next [-] | | Please feel free to cite sources then, otherwise I see no relevancy from you. | | |
| ▲ | danielmarkbruce 3 days ago | parent [-] | | Nathan Lambert wrote a good book recently, and he and his team wrote the paper below about Tulu 3 (Allen Institute). Both are good reads. https://arxiv.org/pdf/2411.15124 | | |
| ▲ | doc_ick 3 days ago | parent [-] | | Thank you for providing an arxiv! An aside, I finally do appreciate single column format now, makes it easier to convert to epub. | | |
| ▲ | danielmarkbruce 3 days ago | parent [-] | | When you are done with the section on RLVR, consider whether the model is predicting tokens, or making moves. There is a reason the word "policy" is used in RL. | | |
| ▲ | doc_ick 2 days ago | parent [-] | | Would still say it’s a token predictor, a fancy one though. I suppose we can agree to disagree. |
|
|
|
| |
| ▲ | 3 days ago | parent | prev [-] | | [deleted] |
|
|
