Remix.run Logo
▲ londons_explore 2 hours ago

> the increasing use of RL

RL, if it works well, could theoretically reduce the perplexity too.

▲ACCount39 2 hours ago | parent [-]

It could indeed, in theory. But in practice, perplexity-as-measured rises due to "controlled mode collapse" nature of RL training.

Things that are useful for base model "next token prediction" can be useless or even actively harmful (should I elaborate?) for CoT, so the model ends up suppressing them under RL pressure. Likewise, things that are useful for CoT can harm base model next token prediction.

But I bring up "perplexity-as-measured", "controlled collapse" and "suppression" because I don't believe that this "ruins" the model's underlying prediction capabilities. Generally, it merely redirects them - brings them more in line with what we care about. It may well be that with a more elaborate training and measurement setup, RL would indeed improve the "perplexity proxy" metric.