Remix.run Logo
▲ seizethecheese an hour ago

The trend is only against a corpus of code. It’s entirely possible that the results are largely due to more intensive pretraining on code. I made a post here of a model training where code proportion is increased and you see corresponding perplexity drop https://micjm.com/blog/lowering-perplexity-without-scaling/

▲Jepacor 13 minutes ago | parent [-]

How do you know it's a corpus of code? I saw nothing describing what exactly their dataset is composed of in the post.

▲seizethecheese 12 minutes ago | parent [-]

Footnote from OpenAI blog post:

3. The primary held-out datasets we’ve been using since GPT-3.5 are sourced from our internal codebase, and those are the numbers reported here (for a fixed snapshot).