A lot of the time you are running experiments and seeing how the data changes, not looking at the data in a vacuum.

Many of the example critiques here don't apply so much when looking at the changes in data:

- if I got 50% more hits on my site this week vs last week, that's meaningful despite 36% of people blocking ads

- if my open rate doubled when I changed my email subject, also meaningful

The other examples are hard to pick holes in as they simply say "Z is a lie", but I can be looking at multiple data sources to decide how much of a lie Z is.

▲

shostack an hour ago | parent | next [-]

Having been at smaller companies without the data, tooling, discipline, and resourcing to conduct viable experiments, and then being at a company that is actually one of the best in the world at it and building solutions for these problems at scale for marketing teams, so many problems come back to the human element in how data is tracked, how teams collaborate or fail to which can lead to pollution and dirty data, and how decisions are made as the moment trade-offs need to be considered it becomes personal and enters messy human relationship territory.

But ultimately from a pure "did this work or not" standpoint you are right. Incrementality experiments are the gold standard.

	▲	specproc 11 minutes ago \| parent [-]
		I think the key point there is "viable experiments", most digital marketing people I've encountered are not into things like deseasoning, controlling, heck, you say "regression" and they blank. Not accounting for trends and covariates would give a misleading impressing even with high quality data.

▲

N_Lens an hour ago | parent | prev [-]

Following this kind of process blindly and optimising for it leads to a terrible product. However it can lead to more revenue in the short term.

	▲	bigbuppo an hour ago \| parent [-]
		It leads to more tracking and higher marketing budgets. And sometimes it leads to more revenue, but since the marketing department is in charge of the statistics, well, did they really?