your perception is that you get more stuff done faster, and that seems to be the conclusion that most using LLM's come to (including myself): but there are no productivity gains that are externally observable to show for it (yet).

That might be because of some lag, but another explanation might be that the tools are really effective at increasing someone's perception of their productivity while net productivity gains measured over long time windows are low or zero. Another explanation might be that the incompetent/negligent users are causing enough harm to nearly neutralize the gains brought by proficient / effective users. I don't know what the explanation is, but there is a mystery in the divergence between users' perceptions of their own productivity, and the observed productivity gains in the economy.

▲

smokedetector1 a day ago | parent | next [-]

I think its that theres a way to use them really well and a way to use them not well and it comes out as a wash in the study if you dont stratify it. Im sorry, call it anecdotal if you want, I dont really care - I know what its doing for me on a day to day basis.

▲

wat10000 a day ago | parent | prev [-]

It would be a weird coincidence if a pretty different way of working turned out to be exactly as productive as the old way, no more and no less. I’m skeptical of these claims. Measuring programmer productivity is notoriously difficult.

▲

mistercheph a day ago | parent [-]

It might be weird, but it might be the result of forces in equilibrium:

e.g. one explanation might be that time spent prompting is time spent not improving, or learning, so it would not be surprising then that the productivity gains might be similar in scale to the productivity lost from learning / improvement that never happens.

another explanation might be that prompting just delays work that will have to be done later when fixing errors. When encountering an error that the LLM doesn't fix on the first attempt the human-in-the-loop doesn't understand the problem very well: they will either pull the slot machine for a while until they hit lucky, or grit their teeth and sit down to understand what the code is doing in the excruciating and boring detail required to find and solve bugs, which is exactly the work that they had offloaded onto the LLM earlier.

	▲	wat10000 a day ago \| parent [-]
		Sure, it’s possible. But I don’t see anything that could constrain those things to be of similar magnitude. If they just happen to cancel each other out by luck, that would be pretty strange.