> They pushed for a retraction ...

That's not right; retractions should only be for research misconduct cases. It is a problem with the article's recommendations too. Even if a correction is published that the results may not hold, the article should stay where it is.

But I agree with the point about replications, which are much needed. That was also the best part in the article, i.e. "stop citing single studies as definitive".

▲

dgxyz an hour ago | parent [-]

I will add it's a little more complicated than I wanted to let on here as I don't identify it in the process. But it definitely was misconduct on this one.

I read the paper as well. My background is mathematics and statistics and the data was quite frankly synthesised.

▲

jruohonen an hour ago | parent [-]

Okay, but to return to replications, publishers could incentivize replications by linking replication studies directly on a paper's website location. In fact, you could even have a collection of DOIs for these purposes, including for datasets. With this point in mind, what I find depressing is that the journal declined a follow-up comment.

But the article is generally weird or even harmful too. Going to social media with these things and all; we have enough of that "pretty" stuff already.

▲

dgxyz an hour ago | parent [-]

Agree completely on all points.

However there are two problems with it. Firstly it's a step towards gamification and having tried that model in a fintech on reputation scoring, it was a bit of a disaster. Secondarily, very few studies are replicated in the first place unless there is a demand for linked research to replicate it before building on it.

There are also entire fields which are mostly populated by bullshit generators. And they actively avoid replication studies. Certain branches of psychology are rather interesting in that space.

	▲	jruohonen 39 minutes ago \| parent [-]
		> Certain branches of psychology are rather interesting in that space. Maybe, I cannot say, but what I can say is that CS is in the midst of a huge replication crisis because LLM research cannot be replicated by definition. So I'd perhaps tone down the claims about other fields.