I feel like the current situation is temporary. LLMs are finding all the bugs. LLMs are also help fixing most of the bugs. Once most of the bugs are fixed, LLMs should be good at finding bugs before shipping them, the stream of bug reports will die down, and we'll be back to vulnerabiltiy reports being special.

Further, the fact that bugs are so easy to find by LLMs means there is strong incentives to find ways to minimize creating bugs in the first place. That could be new or better languages, less 3rd party dependencies, more vetted code, better linters, better fuzzers, whatever. The point the new reality of bugs being easy to find will, actually must, lead to less bugs eventually because the world can't function with easy to find bugs.

"Temporary" can be an awfully long time. There is ample evidence that discovery rate of bugs (many of which can be bucketed into vulnerabilities) in any non-trivial piece of software is more or less stable.[0] In a recent podcast episode the ex-CISO of Adobe commented that every now and then they'd take a sustained squeeze to find all occurrences of a given type of bug (ie. source of vulnerability) in a codebase. They'd find a good amount of them and fix them.

Then a year or two later they'd repeat the operation and they'd find about the same amount of same types of bugs. In many occasions in code that had been in place in the previous round and had remained essentially untouched.

Paraphrasing what the Gruqg has quipped - a large piece of software has infinity bugs. Infinity minus N is still infinity.

0: Discovery rate with regards to the time spent looking for bugs. LLM-powered bug hunting has amped up the speed with which code bases can be investigated.

I feel this sentiment is wishful thinking,but I want to start by saying I hope it turns out to be correct.

I find that often bugs will be created when using an LLM, like others have said. Saying that this can then be fixed by identifying all the bugs created by an LLM with an LLM doesnt guarantee another bug is not introduced when the LLM is addressing the initial problem.

Also, what if the LLM has a blind spot. They certainly also could be incapable of finding or fixing a bug. They dont pass any benchmark at 100% right now. Also also, guaranteeing there are no bugs in your code is like saying you have 100% test coverage, all of the tests pass, and they are written perfectly. Saying that you can simply identify and fix the bugs also assumes there is enough time and energy to find all of the bugs that exist within a project and then to address them. Even LLMs use time and energy. In a sufficiently complex system that is certainly wishful thinking.

Considering the size and complexity of a lot of modern software (like web browsers, 3d modelling software, game engines, etc.) software is just too complex to not have bugs even when created and managed by LLMs.

There will continue to be bugs in code and we will simply have to live with the fact that LLMs make it easier to exploit computer systems. I mean consider a hardware bug like Spectre [0]. If bugs like this become easier to find does that mean our existing hardware will just become obsolete more quickly? that type of problem can be addressed, but at quite a high cost.

Not sure what all of this means for the future.

0. https://en.wikipedia.org/wiki/Spectre_%28security_vulnerabil...

That supposes that LLMs can write secure software. Also, if we assume that finding bugs is easier that not creating them (reasonable I would say), the supply of bugs will never be exhausted.

thats is definitely NOT what the article says.

Are you making a counterpoint that the reports are so good and must all be addresses, but the problem is "llm finding all the bugs" so fast us poor slow humana cannot keep up?

because if so, i suggest you write a new article.

Lol you think LLMs are generating bug free code?