It’s a very hard experiment to run. You have a population that’s already “treated”. You can’t blind them to the fact that they’re using AI tools. It’s hard to imagine a study that wouldn’t have serious flaws that people would then use to dismiss and form their own conclusions. Sure you have METR but that was very low n with a very old model.

I think the surest sign of productivity gains is the sheer volume of adoption. If you look beyond headlines, adoption is just incredible. Of course adoption does not necessarily point to productivity gains, but if this was some sort of FOMO or smoke and mirrors you would not see this much retention and this feverish a pace of adoption. You would not see a large segment of the profession using coding agents exclusively. All of these companies track productivity, again with imperfect proxies, yet everything points to a pretty consistent picture. Same with benchmarks, again a lot of crappy benchmarks but a lot of high quality ones too and a very diverse collection of tasks and capabilities they probe.

▲

48terry 4 hours ago | parent [-]

Your second paragraph appears to be 3 different instances of saying "X does not necessarily point to productivity gains... but in the case of AI, X definitely means productivity" without really saying why that is true or why other explanations do not fit.

Adoption meaning productivity supposes there are no other dominant factors for the AI push nor AI retention. It is possible for practices to be picked up or continued in spite of causing productivity DROPS. What studies have suggested are factors that make for productive work environments and what is actually enforced in the workplace are different things.

	▲	aspenmartin 4 hours ago \| parent [-]
		It’s 3 different weak but complimentary proxies. We form beliefs from imperfect evidence and I find these fairly convincing when it’s hard to find any hard evidence of no productivity and exactly the scenario you would expect under the hypothesis that we do see productivity gains. None of this is supposed to be unassailable. I would challenge then if you disagree what the evidence you have for this is? Adoption implying at least some significant productivity gains doesn’t contradict there being other factors. You’re seeing entire companies reshaped. The argument is this is all for show or CEOs are in some sort of idiot class? “It is possible for practices to be picked up or continued in spite of causing productivity drops” well of course. I just find that incredibly far away from Occam’s razor. My point is: we have lots of evidence that’s highly consistent with real productivity gains, and I don’t see many pieces of evidence to the contrary.