| ▲ | seizethecheese an hour ago | |||||||
The trend is only against a corpus of code. It’s entirely possible that the results are largely due to more intensive pretraining on code. I made a post here of a model training where code proportion is increased and you see corresponding perplexity drop https://micjm.com/blog/lowering-perplexity-without-scaling/ | ||||||||
| ▲ | Jepacor 13 minutes ago | parent [-] | |||||||
How do you know it's a corpus of code? I saw nothing describing what exactly their dataset is composed of in the post. | ||||||||
| ||||||||