Remix.run Logo
mike_hearn a day ago

Replication is overrated. There's no point in doing more of it. You'd just replicate a ton of bad science:

https://blog.plan99.net/replication-studies-cant-fix-science...

N.B. nobody needed to replicate this study. It only took basic sanity checks to spot that it was very unlikely to be true (and thus, very unlikely to replicate). And they went much further than needed. They could have stopped at "the claimed effect is much larger than the height difference between men and women". This obviousness is a common pattern:

https://www.fantasticanachronism.com/p/whats-wrong-with-soci...

> Prediction markets work well because predicting replication is easy. There's no need for a deep dive into the statistical methodology or a rigorous examination of the data, no need to scrutinize esoteric theories for subtle errors—these papers have obvious, surface-level problems.

So you could also say, how do we change the incentives to get people to peer review properly? That would be a lot cheaper.

The obvious solution is for people to get paid to audit work professionally, and be fired if they sign off on bad results. But that exists already, it's called a corporate lab. Managers are incentivized by carrots and sticks to properly understand what their researchers are doing and if anyone is doing fraud they will be fired, maybe prosecuted. Universities don't do this because so much of what they do exists purely to spend money, with no higher purpose. There's no specific goal or outcome they're trying to achieve, and so no reason to do unpleasant things like enforcing incentives to get there.

So in the end the solution is just to defund university research. The social structures are fundamentally unfixable. You'll never get people to care if the only purpose of their job's existence is to convince voters governments are funding science.