Incredibly hard problem, but METR had a good method. They had people estimate how long a task would take (before knowing whether AI would be used), and then randomly assigned each task to “with AI” or “without AI.” When the data was in, they compared actual/estimate ratios of the two populations.

(Presumably, they used a t-test that only compared people against themselves.)

Interestingly, for that study (released in 2025), participants self-rated themselves as 20% more productive, but were measured as being 19% less productive.