It does seem like its way too easy to do this kind of fraud.

The study seems extremely easy to replicate (no fancy equipment needed, just ask some people to proofread), and yet nobody did until now. Is there some way to change incentives to get people to try and replicate? Assign undergrads to replicate random research papers?

Replication is unpaid QA on someone else's product. Nobody gets tenure for finding the bug, hence the field has been meaning to get around to it for 20 years.

> The study seems extremely easy to replicate (no fancy equipment needed, just ask some people to proofread), and yet nobody did until now

They were just putting it off.

My proposal, once you complete your study, if you find a significant positive result you should be required (prior to publication) to describe your methodology in full, and have another lab (who you never have any contact with) blindly take your methodology, and repeat it (on video), and then do their own independent analysis of the data.

It's okay if it's $200k or so to do each of these, the university should pay. The point is if this methodology were applied people would have incentive to cheat anymore. It would also serve as an immediate replication.

Schools won't do this because they have a perverse incentive for big successes and quiet retractions. It's corruption all the way up.

Why do we even accept anything as true without replication by default?

Replication is overrated. There's no point in doing more of it. You'd just replicate a ton of bad science:

https://blog.plan99.net/replication-studies-cant-fix-science...

N.B. nobody needed to replicate this study. It only took basic sanity checks to spot that it was very unlikely to be true (and thus, very unlikely to replicate). And they went much further than needed. They could have stopped at "the claimed effect is much larger than the height difference between men and women". This obviousness is a common pattern:

https://www.fantasticanachronism.com/p/whats-wrong-with-soci...

> Prediction markets work well because predicting replication is easy. There's no need for a deep dive into the statistical methodology or a rigorous examination of the data, no need to scrutinize esoteric theories for subtle errors—these papers have obvious, surface-level problems.

So you could also say, how do we change the incentives to get people to peer review properly? That would be a lot cheaper.

The obvious solution is for people to get paid to audit work professionally, and be fired if they sign off on bad results. But that exists already, it's called a corporate lab. Managers are incentivized by carrots and sticks to properly understand what their researchers are doing and if anyone is doing fraud they will be fired, maybe prosecuted. Universities don't do this because so much of what they do exists purely to spend money, with no higher purpose. There's no specific goal or outcome they're trying to achieve, and so no reason to do unpleasant things like enforcing incentives to get there.

So in the end the solution is just to defund university research. The social structures are fundamentally unfixable. You'll never get people to care if the only purpose of their job's existence is to convince voters governments are funding science.

"easy to replicate (no fancy equipment needed, just ask some people to proofread)"

? This was a human subject study, that would cost $$ and there would be very little upside to confirmation and only a little notoriety with a failure to replicate (which by the way is not dispositive, people might need multiple failed replications to know who to believe, depending on the size of the replication). Maybe you didn't read the post: this was data fraud discovered by statistical irregularities. That is indeed becoming easy to spot with more widespread statistical literacy, at least for unsophisticated jobs like this from decades ago.

> Is there some way to change incentives to get people to try and replicate?

Sure. Pay them to do it.

Fund it. Unfortunately there's hardly any money for regular studies, let alone replication studies. If you want better science there has to be tax dollars behind it. Otherwise corporations choose what gets funded, and they only fund narratives and experiments they think will be profitable for them.

[dead]