Right, but you are just relying on a different form of random, there. The whole point of making controls and then building experiments on changing them, is to get more power from fewer observations. No?

Again, it is off to think that one is automatically superior to the other. Certainly to the exclusion of the other. And that is what feels off with the framing of the parent post. I am perfectly fine saying you should use both observational and controlled trials. But I think it is also wrong to think you don't have to build experiments to test interventions.

This is why you put metrics in your service code. So that you can observe them behave and look for things to change. This is also why you do test cases on your code, so that you can specifically target your change.

Now, I fully back the idea that just A/B testing something doesn't automatically mean you learn something true. But neither does observing a strong outcome on uncontrolled data.

Yeah, GP is basically saying:

"Large controlled experiments are costly and can hurt people who opt-in to informed consent. Instead, we should do significantly, significantly larger experiments, with undefined success/failure conditions, and no informed consent."

Insane opinion