Elephant in the room : how many of these bugs were written by LLMs in the first place ?

Because creating 100x more bugs and fixing 100x more isn't something to be proud of.

Cthulhu_ 9 hours ago | parent | next [-]

Can't have been much, as the Chrome project is 20+ years old and LLMs have only recently become a thing; they did not slacken their code review and testing practices with the advent of LLM code generators. And I doubt they do much development on the affected areas at the moment, they mention a 13 year old issue for example.

But it's an open source project, you can go and figure out whether your assertion is correct.

▲

lbrito 2 hours ago | parent | next [-]

>Can't have been much, as the Chrome project is 20+ years old and LLMs have only recently become a thing

That's not a great argument. GPT3.5 was launched over 4 years ago, so that's already significant compared to Chrome's age. LLMs notoriously also produce more verbose code than a human, so its not out of question that it might produce 5x more bugs on average given the same time frame.

▲

goldenarm 9 hours ago | parent | prev [-]

The single 13yo issue is anecdata.

They obviously have the full git blame statistics but chose not to include them in the blog post, which is a bit concerning.

	▲	rpdillon an hour ago \| parent [-]
		It's weird to assume that this is all smoke and mirrors where AI is fixing AI's bugs. When we dove into the rsync issue, what was happening is that AI was revealing bugs that had been previously unknown, resulting in the need to fix them. I would be stunned if that were not the case here.

▲

tarkin2 8 hours ago | parent | prev | next [-]

If people are handing off the coding to AI more and more then they'll be less skilled at spotting potential bugs in the first place. AI, course of, could be run to find potential bugs in the AI-coded features before commit. But then you're getting further and further towards a world where code, and code that runs infrastructure, is incomprehensible. We'll have to start sacrificing the first-born to the silicon in the hopes of a good build for code that controls your electrical sub-station.

▲

surajrmal 4 hours ago | parent | prev | next [-]

Look at the git stats for the project. Lines of code submitted haven't changed dramatically. Not everyone is just merging slop into the codebase. There are many ways to use AI, and most established players are not merging vibe coded nonsense.

▲

llm_nerd 8 hours ago | parent | prev | next [-]

That is a stat you invented out of thin air, based upon nothing but pejorative hope, against all logic or rational consideration of the timeline or the scale of the project, and you declare it the "elephant in the room"?

HN is so weird on the topic of AI, and so many desperately are trying to contrive a reality. Odd stuff.

▲

goldenarm 8 hours ago | parent | next [-]

It's not an invention it's a question. If the number is <20% that's great news for the Chrome team.

▲

llm_nerd 5 hours ago | parent [-]

The elephant in the room is how many people in this discussion are actually alien lifeforms.

The elephant in the room is how many turtles obtained law degrees.

The elephant in the room is whether this is worth discussing because we're all in a simulation.

No, it isn't the elephant in the room. You made up some horseshit that betrays how little you know about the field, which will get upvoted because HN has a hearty contingent of "if I pretend AI is actually useless, maybe that will manifest in reality" zealots. It's bizarre.

	▲	goldenarm 5 hours ago \| parent [-]
		Turtles with law degrees don't exist, but LLMs that increase tech debt in code bases do. Maybe recent ones got better, and it's useful to quantify the progress being made.

▲

xxs 4 hours ago | parent | prev [-]

this is really not reddit

▲

buzzin__ 7 hours ago | parent | prev [-]

If you assume that AI can produce code with less bugs, your 100 times increase in fresh bugs means you also increased your speed for new functionality by more than 100 times.

If models can spot 13 years old critical bugs, I think they can also produce fresh code without those bugs. The skillset is the same.

	▲	zahlman an hour ago \| parent [-]
		> If models can spot 13 years old critical bugs, I think they can also produce fresh code without those bugs. The skillset is the same. As if humans are incapable of spotting critical bugs simply because they've existed for a long time? And we know humans don't produce fresh bug-free code, because the old bugs exist.