The fraction of A/B tests I've seen personally that mentioned ANOVA at all is very small. Or thought that critically about experiment design. Understanding of p values is also generally poor; prob/stat education in engineering and business degrees seems to be the least-covered-or-respected type of math.

Even at places that want to ruthlessly prioritize velocity over rigor I think it would be better to at least switch things up and worry more about effect size than p-value. Don't bother waiting to see if marginal effects are "significant" statistically if they aren't significant from the POV of "we need to do things that can 10x our revenue since we're a young startup."

▲

fho 8 hours ago | parent | next [-]

> mentioned ANOVA at all is very small

That's because nobody learns how to do statistics and/or those who do are not really interested in it.

I taught statistics to biology students. Most them treated the statistics (and programming) courses like chores. Out of 300-ish students per year we had one or two that didn't leave uni mostly clueless about statistics.

▲

TeMPOraL 8 hours ago | parent [-]

FWIW, universities are pitching statistics the same way as every other subject, i.e. not at all. They operate under a delusion that students are deaperately interested in everything and grateful for the privilege of being taught by a prestigious institution. That may have been the case 100 years ago, but it hasn't been for decades now.

For me, stats was something I had to re-learn years after graduating, after I realized their importance (not just practical, but also epistemological). During university years, whatever interest I might have had, got extinguished the second the TA started talking about those f-in urns filled with colored balls.

▲

fho 5 hours ago | parent | next [-]

Also part of the problem:

> those f-in urns filled with colored balls.

I did my Abitur [1] in 2005, back then that used to be high school material.

When I was teaching statistics we had to cut more and more content from the courses in favor of getting people up to speed on content that they should have known from school.

[1] https://en.m.wikipedia.org/wiki/Abitur

▲

yusina 7 hours ago | parent | prev | next [-]

And you didn't have the mental capacity to abstract from the colored balls to whatever application domain you were interested in? Does everything have to come pre-digested for students so they don't have to do their own thinking?

	▲	stirfish 6 hours ago \| parent [-]
		Hey yusina, that's pretty rude. What's a different way you could ask your question?

▲

fn-mote 7 hours ago | parent | prev [-]

> They operate under a delusion that students are desperately interested in everything

In the US, students are the paying customers. The consequence for not learning everything is lowered skills available for the job market (engineering) or life (philosophy?).

To me it is preferable that students who do not understand are not rated highly by the university (=do not get top marks), but “forcing” the students to learn statistics? That doesn’t make much sense.

Also, there’s nothing wrong with learning something after uni. Every skill I use in my job was developed post-degree. Really.

▲

enaaem 5 hours ago | parent | prev [-]

Instead of trying to make p-values work. What if we just stopped teaching p-values and confidence intervals, and just teach Bayesian credible intervals and log odds ratios? Are there problems that can only be solved with p-values?