It's actually based on the results of three surveys conducted by two different parties. While surveys are subject to all kinds of biases and the gains are self-reported, their findings of 25% - 33% producitivity do match the gains shown by at least 3 other randomized studies, one of which was specifically about programming. Those studies are worth looking at as well.

foolswisdom 7 days ago | parent [-]

It's worth noting that the METR paper that found decreased productivity also found that many of the developers thought the work was being sped up.

	▲	keeda 6 days ago \| parent [-]
		Yes, self-reporting has biases and estimating tasks is still a fool's errand, which is why I noted that the estimates from these surveys matched the findings from other RTC studies. However, what doesn't get discussed enough about the METR study is that there was a spike in overall idle time as they waited for the AI to finish. I haven't run the numbers so I don't know how much of the increased completion time it accounts for, but if your cognitive load drops almost to 0, it will of course feel like your work is sped up, even though calendar time has increased. I wonder if that is the more important finding of that paper.