Remix.run Logo
▲ x313 2 hours ago

I thought perplexity on simple text data has been saturated for a while (as in perplexity is uncorrelated to task success - ex. https://arxiv.org/html/2410.23771v2).

I suspect what's happening is their test dataset is actually just something really hard, like math papers or something. In that case perplexity would just be a direct measure of math knowledge, which means this blog would just be saying that better math knowledge predicts better math skills.

▲seizethecheese 2 hours ago | parent [-]

The test dataset is a code repository, it says so in footer.

I had the same thought and just started writing something about how maybe they just added more code to pretraining corpus over time.

Edit: here it is https://micjm.com/blog/lowering-perplexity-without-scaling/