Right, but that cost is also incurred by human-written code that happens to have bugs.

In theory experienced humans introduce less bugs. That sounds reasonable and believable, but anyone who's ever been paid to write software knows that finding reliable humans is not an easy task unless you're at a large established company.

▲

MeetingsBrowser 3 hours ago | parent | next [-]

The question then becomes, can LLMs generate code close to the same quality as professionals.

In my experience, they are not even close.

▲

mathgeek 2 hours ago | parent [-]

We should qualify that kind of statement, as it’s valuable to define just what percentile of “professional developers” the quality falls into. It will likely never replace p90 developers for example, but it’s better than somewhere between there and p10. Arbitrary numbers for examples.

▲

MeetingsBrowser an hour ago | parent [-]

Can you quantify the quality of a p90 or p10 developer?

I would frame it differently. There are developers successfully shipping product X. Those developer are, on average, as skilled as necessary to work on project X. else they would have moved on or the project would have failed.

Can LLMs produce the same level of quality as project X developers? The only projects I know of where this is true are toy and hobby projects.

	▲	mathgeek an hour ago \| parent [-]
		> Can you quantify the quality of a p90 or p10 developer? Of course not, you have switched “quality” in this statement to modify the developer instead of their work. Regarding the work, each project, as you agree with me on from your reply, has an average quality for its code. Some developers bring that down on the whole, others bring it up. An LLM would have a place somewhere on that spectrum.

▲

verdverm 3 hours ago | parent | prev [-]

There was a recent study posted here that showed AI introduces regressions at an alarming rate, all but one above 50%, which indicates they spend a lot of time fixing their own mistakes. You've probably seen them doing this kind of thing, making one change that breaks another, going and adjusting that thing, not realizing that's making things worse.