> Wasn’t there a study that said that using LLMs makes people feel more productive while they actually are not?

On a tangent; that study is brought up a lot. There are some issues with it, but I agree with the main takeaway to be weary of the feeling of productivity vs actual productivity.

But most of the time its brought up by AI skeptics, that conveniently gloss over the fact it's about averages.

Which, while organizationally interesting, is far less interesting than to discover what is and isn't currently possible at the tail end by the most skillful users.

The key insight from the study is that even the users that did see an increase in productivity overestimated that increase.

Taken along with the dozens of other studies that show that humans are terrible at estimating how long it will take them to complete task, you should be very skeptical when someone says an LLM makes them x% more productive.

There’s no reason to think that the most skillful LLM users are not overestimating productivity benefits as well.

Engineers have always been terrible at measuring productivity. Building a new internal tool or writing a bunch of code is not necessarily productive.

Productivity is something that creates business value. In that sense an engineer who writes 10 lines of code but that code solves a $10M business problem or allows the company to sign 100 new customers may be the most productive engineer in your organization.

Not to mention the study doesn't really show a lack of productivity and they include some key caveats in it outlining how they think productivity increases using LLMs