Remix.run Logo
▲ seizethecheese 2 hours ago

The test dataset is a code repository, it says so in footer.

I had the same thought and just started writing something about how maybe they just added more code to pretraining corpus over time.

Edit: here it is https://micjm.com/blog/lowering-perplexity-without-scaling/