This seems kind of crazy. If LLMs are so stunningly good at finding vulnerabilities in code, then shouldn't the solution be to run an LLM against your code after you commit, and before you release it? Then you basically have pentesting harnesses all to yourself before going public. If an LLM can't find any flaws, then you are good to release that code.

A few years ago, I invoked Linus's Law in a classroom, and I was roundly debunked. Isn't it a shame that it's basically been fulfilled now with LLMs?

https://en.wikipedia.org/wiki/Linus%27s_law

After a release, attackers have effectively infinite time to throw an LLM against every line of your code - an LLM that only gets smarter and cheaper to run as time passes. In order to feel secure you’d need to do all the work you’d imagine an attacker would ever do, for every single release you ship.

As LLMs improve and adoption grows, maintaining a FOSS project is becoming more complex and more expensive in terms of time and manpower. That part is easy to understand.

It is also become a trend that LLM-assisted users are generating more low-quality issues, dubious security reports, and noisy PRs, to the point where keeping the whole stack open source no longer feels worth it. Even if the real reason is monetization rather than security, I can still understand the decision.

I suspect we will see more of this from commercial products built around a FOSS core. The other failure mode is that maintainers stop treating security disclosures as something special and just handle them like ordinary bugs, as with libxml2. In that sense, Chromium moving toward a Rust-based XML library is also an interesting development.

LLMs really are stunningly good at finding vulnerabilities in code, which is why, with closed-source code, you can and probably will use them to make your code as secure as possible.

But you won't keep the doors open for others to use them against it.

So it is, unfortunately, understandable in a way...

Write simple code. Do what you said, which is a very good idea. Test LLM security against the compiler too.

Every change would introduce the possibility of a vulnerability being added to the system and one would need to run the LLM scan across the entire code base. It gets very costly in a environment where you are doing regular commits. Companies like Github already provide scanning tools for static analysis and the cost is already high for them.

Attackers only need LLMs to be good at randomly finding one vulnerability, whereas service providers need them to be good at finding all such vulnerabilities.

That’s a non-trivial cost for commonly severely underfunded open source projects

I mean, you should definitely have _some_ level of audit by LLMs before you ship, as part of the general PR process.

But you might need thousands of sessions to uncover some vulnerabilities, and you don’t want to stop shipping changes because the security checks are taking hours to run

It's entirely possible to address all the LLM-found issues and get an "all green" response, and have an attacker still find issues that your LLM did not. Either they used a different model, a different prompt, or spent more money than you did.

It's not a symmetric game, either. On defense, you have to get lucky every time - the attacker only has to get lucky once.

[dead]