Remix.run Logo
abeppu 2 days ago

This seems sketchy in that they're giving themselves N*M opportunities to find significant correlations, which they acknowledge -- but with a bias that a significant correlation is not expected to be spurious if it aligns with a claim previously made elsewhere.

> Some of the additional statistically significant findings (e.g., the perception of Rubin’s Vase and openness, the Horse-Seal and intuitive decision making, and the Duck-Rabbit and extraversion and conscientiousness) appear somewhat isolated, are not related to previous research, or claims being made in social media posts and websites. As such, they may be the result of multiple analyses.

I know there's a body of work on "False Discovery Rate", and I think it would be more appropriate to use some of those tools to directly adjust for the number of attempts they're giving themselves.

nerdponx 2 days ago | parent | next [-]

Correct, FDR control procedures have been part of the statistics literature for at least 50 years, and are relatively easy to implement by hand. There's no excuse.

aqueueaqueue 2 days ago | parent | prev [-]

Is this the green jelly beans xkcd?

nerdponx a day ago | parent [-]

Yes, and it's a legitimate problem in science. In principle (not that anyone does this), you should probably apply some kind of FDR or FWER control procedure to the entire body of literature in each field. And that's before you get into the known effect of publication bias.