▲ | travisjungroth 7 hours ago | |
> If you have a big effect you’ll see it even with small data. That’s in line with what I was saying so I’m not sure where I missed the point. P-value a function of effect size, variance and sample size. Bigger wins would be those that have a larger effect and more consistent effect, scaled to the number of users (or just get more users). | ||
▲ | bigfudge an hour ago | parent [-] | |
> But in most cases, at a startup, you should be going after wins that are way more impactful and end up having p-values lower than 0.05, anyway. This was the part I was quibbling with. The size of the p value is pretty much irrelevant unless you know how much data you are collecting. The p values might always be about ~.05 if you know the effects are likely large and powered the study appropriately. |