| ▲ | londons_explore 2 hours ago | |
> the increasing use of RL RL, if it works well, could theoretically reduce the perplexity too. | ||
| ▲ | ACCount39 2 hours ago | parent [-] | |
It could indeed, in theory. But in practice, perplexity-as-measured rises due to "controlled mode collapse" nature of RL training. Things that are useful for base model "next token prediction" can be useless or even actively harmful (should I elaborate?) for CoT, so the model ends up suppressing them under RL pressure. Likewise, things that are useful for CoT can harm base model next token prediction. But I bring up "perplexity-as-measured", "controlled collapse" and "suppression" because I don't believe that this "ruins" the model's underlying prediction capabilities. Generally, it merely redirects them - brings them more in line with what we care about. It may well be that with a more elaborate training and measurement setup, RL would indeed improve the "perplexity proxy" metric. | ||