| ▲ | x313 2 hours ago | |
I thought perplexity on simple text data has been saturated for a while (as in perplexity is uncorrelated to task success - ex. https://arxiv.org/html/2410.23771v2). I suspect what's happening is their test dataset is actually just something really hard, like math papers or something. In that case perplexity would just be a direct measure of math knowledge, which means this blog would just be saying that better math knowledge predicts better math skills. | ||
| ▲ | seizethecheese 2 hours ago | parent [-] | |
The test dataset is a code repository, it says so in footer. I had the same thought and just started writing something about how maybe they just added more code to pretraining corpus over time. Edit: here it is https://micjm.com/blog/lowering-perplexity-without-scaling/ | ||