| ▲ | seizethecheese 2 hours ago | |
The test dataset is a code repository, it says so in footer. I had the same thought and just started writing something about how maybe they just added more code to pretraining corpus over time. Edit: here it is https://micjm.com/blog/lowering-perplexity-without-scaling/ | ||