Remix.run Logo
▲ WASDx 2 hours ago

Do they say that they measure perplexity at the pretraining phase? I thought most of the advancements have come from RL during posttraining. Would an older model with modern posttraining have the same perplexity while being significantly better? Then I don't see it as a meaningful metric.

▲reasonableklout 2 hours ago | parent | next [-]

A common view is that RL is eliciting behaviors learned in pretraining [1], so even though many recent gains are achieved via posttraining, pretraining is still extremely important because base models provide all the latent potential. This is discussed a bit in the OP.

[1]: https://www.interconnects.ai/p/elicitation-theory-of-post-tr...

▲famouswaffles 2 hours ago | parent | prev [-]

They are measuring the perplexity of pre-trains.