Ironic that "Reinvent science" would publish such a trash article. Did they even read the original paper? This is not at all what it says.

https://arxiv.org/pdf/2305.13169

The "poisonous scientific literature" removal is on page 14. Look at the table. Removing academic pieces changes performance by on average 0.44%! You could sneeze and change the performance by that much. You could rerun with a different seed and change the performance by more than that. You could retrain on a different GPU that orders floating point operations differently and change performance by that much. etc. This is meaningless.

Also, the authors explain exactly why this happens! It's on that page even. The academic data hurts a little on datasets which aren't academic. It hurts on common sense reasoning like SocialIQA (Q: "Jordan wanted to tell Tracy a secret, so Jordan leaned towards Tracy. Why did Jordan do this?" A: "Make sure no one else could hear"). Shocking that you can't learn this from academic publications.

This is just substandard blog slop that give science a bad name.

The scary part is: "Dan Recht and Ben Reinhardt trained as scientists and now coach scientists. They do other work but don’t link to it here." As someone who has advised plenty of PhD students I can't find the words to express my disdain at that line and these jokers.