| ▲ | noodletheworld 19 hours ago | |
There are many resources that will explain this rigorously if you search for the term “p-hacking”. The TLDR as I understand it is: All data has patterns. If you look hard enough, you will find something. How do you tell the difference between random variance and an actual pattern? It’s simple and rigorously correct to only search the data for a single metric; other methods, eg. Bonferroni correction (divide p by k) exist, but are controversial (1). Basically, are you a statistician? If not, sticking to the best practices in experimentation means your results are going to be meaningful. If you see a pattern in another metric, run another experiment. | ||