> the chance at least one is noise is 1 – 0.95⁵ ≈ 23 %

Yes, but that's not really the big deal that you're making it out to be, since it's (usually) not an all-or-nothing thing. Usually, the wins are additive. The chance of each winner being genuine is still 95% (assuming no p-hacking), and so the expected number of wins out of those 5 will be be 0.95 * 5 = 4.75 wins (by linearity of expectation), which is a solid win rate.

▲

kgwgk 6 months ago | parent | next [-]

>> the chance at least one is noise is 1 – 0.95⁵ ≈ 23 %

> The chance of each winner being genuine is still 95%

Not really. It depends on what’s the unknown (but fixed in a frequentist analysis like this one) difference between the options - or absence thereof.

If there is no real difference it’s 100% noise and each winner is genuine with probability 0%. If the difference is huge the first number is close to 0% and the second number is close to 100%.

	▲	akoboldfrying 6 months ago \| parent [-]
		You're right, I made the classic frequentist statistics mistake. All we can talk about with certainty are the probabilities of things happening under a specific null model.

▲

ec109685 6 months ago | parent | prev [-]

Good point. The 23% in the example refers to the worst case where 5 tests are all null throughout the period.