▲ | majormajor 8 hours ago | |||||||||||||||||||||||||||||||||||||
The fraction of A/B tests I've seen personally that mentioned ANOVA at all is very small. Or thought that critically about experiment design. Understanding of p values is also generally poor; prob/stat education in engineering and business degrees seems to be the least-covered-or-respected type of math. Even at places that want to ruthlessly prioritize velocity over rigor I think it would be better to at least switch things up and worry more about effect size than p-value. Don't bother waiting to see if marginal effects are "significant" statistically if they aren't significant from the POV of "we need to do things that can 10x our revenue since we're a young startup." | ||||||||||||||||||||||||||||||||||||||
▲ | fho 6 hours ago | parent | next [-] | |||||||||||||||||||||||||||||||||||||
> mentioned ANOVA at all is very small That's because nobody learns how to do statistics and/or those who do are not really interested in it. I taught statistics to biology students. Most them treated the statistics (and programming) courses like chores. Out of 300-ish students per year we had one or two that didn't leave uni mostly clueless about statistics. | ||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||
▲ | enaaem 4 hours ago | parent | prev [-] | |||||||||||||||||||||||||||||||||||||
Instead of trying to make p-values work. What if we just stopped teaching p-values and confidence intervals, and just teach Bayesian credible intervals and log odds ratios? Are there problems that can only be solved with p-values? |