| ▲ | WASDx 2 hours ago | |
Do they say that they measure perplexity at the pretraining phase? I thought most of the advancements have come from RL during posttraining. Would an older model with modern posttraining have the same perplexity while being significantly better? Then I don't see it as a meaningful metric. | ||
| ▲ | reasonableklout 2 hours ago | parent | next [-] | |
A common view is that RL is eliciting behaviors learned in pretraining [1], so even though many recent gains are achieved via posttraining, pretraining is still extremely important because base models provide all the latent potential. This is discussed a bit in the OP. [1]: https://www.interconnects.ai/p/elicitation-theory-of-post-tr... | ||
| ▲ | famouswaffles 2 hours ago | parent | prev [-] | |
They are measuring the perplexity of pre-trains. | ||