It very much depends on how you generate it, from prompting to test suites, but most importantly, your attitude. The extra security costs you time and money.

There are still folks, many of them, who think that time to market is everything, and anything beside it should be ignored. "It compiles? Ship it!" is a phrase from 1990s; LLMs just make this approach much easier. We will have plenty of machine-generated software with the security posture of a colander hit by a buckshot.

Physician heal thyself?

Tell an LLM to break into its own system as part of a feed back loop, but with no context of the initial creation? It should iteratively get to 'safe' shouldn't it (subject to the tokens you want to spend)?

It is not just code, it is also deploying Software. I constantly have to correct or steer the 'ai' in conversations towards safer directions or better following even the most basic security principles. Here I feel it is still more the overambitious intern that needs to be carefully overseen, but has a lot of time and energy at hand.

Yeah they are pretty bad at it still, latest models tried to ship a basic SQL injection flaw. Was also happy to build out an xss flawed module end-to-end. Definitely have to keep a close eye on it.

My vibe code friends have pretty much infinite security issues. Don't even have basic data modeling at client-server boundaries or anything. Also had Google analytics embed that read and saved everyone's credit card details as clear text in analytics.

Seems to make pretty much infinite mistakes if you don't take care and tell it what not to do or how to build certain features.

What models are people actually using to pump out code? That is the relevant comparison, not the absolute best in class models.

It's been shown to be bad at secure coding _unless_ you specifically prompt models to watch out for common vulnerabilities, at which point they're actually quite good at containing the issues, though the studies here are somewhat older (eg https://arxiv.org/abs/2211.03622).

The issue then is that most people aren't actually good at keeping an eye out for security issues in code proactively, so things slip through the cracks (whether human-written or not), just at a much accelerated pace.

like most software, do you assume people are driving lamborghinis, or the cheapest make and model they can find...

But if AI produces code with 90% fewer vulnerabilities per line, but generates 10 times as much code, the total security exposure remains the same .

I know you're joking - but I think it will remain a challenge.

Quality and Security are emergent properties of software - especially when considering software-as-deployed in the real world.

And the problems with "good taste" and good architecture and data modelling - will probably leave ai code open to a lot of issues (Claude code cli issues is a great example of how bad this can get).

Having good architects in charge can help - but it requires a lot of discipline to keep the feature set small so that maintaining quality and security become feasible.

It _is_ wonderful. You're never required to adopt software engineering best practices, but eventually everyone does because either it makes our lives easier or code better.

Having tests / CI pipelines, etc are the same thing? Now you can choose to add a cheap step in your pipeline checking if e.g. some engineer did something stupid like allow SQL injections.

The floor of software quality once it gets to review is permanently raised.

Dystopian days, where you will be forced to write some syntax, that is then handed off to a an Ai tester. The tester will return it to you ready for handing the code to three frontier Ai's. The judges of code.

They approve your code, gold, if not; back to the coding machine and minus 1000 human tokens. This will be done via the implants of brain chips

When you know you're looking for an exploit, it's easily verifiable and you can iterate fairly rapidly and get fast feedback, you can just have a test that you don't get a 403 back or something. LLMs, in general, seem good at problems like this.

When you're developing, there's a ton of degrees of freedom, and security encompasses only some of them. To be sure, if you set up a quickly verifiable test for every security flaw, you could probably get LLMs to satisfy them somehow, but there's a lot of those, and it's not necessarily practical.

I have the same experience. Most of articles I read was reducing it to "just a tool, nothing more"

The AI just accelerated the discovery. The issues have existed for years and many major versions.

It's not stupid if it works.

Concretely, I would like to see it benchmarked:

Have Opus 5.5 implement the authentication, with a 6.1 Sol reviewer. Out of 100 times, how often is the result insecure?

I wouldn't be surprised if it's zero times.

There is also the possibility of specifically prompting for a security audit on top, which would perhaps further lower the risk of overlooked issues.

Now one can say that due to the non-deterministic nature of the tools, there is always a non-zero chance of producing insecure implementation.

That's not really important here - obviously manual review isn't deterministic either, the risk will never be zero.

> Not only this, a significant number of developers have stopped looking at code

This is what I find frustrating about the whole AI/slop debate. Developers stopped looking at code a long time ago... Well before LLMs. Take for example a typical nodejs project with its 100s of dependencies. Is a developer going to look through the layers upon layers of code running their app? Of course not, there's to much of it. We've seen supply chain attacks even this week.

Actually what I think will happen is that all those abstractions, dependencies will become irrelevant now that an LLM can just write custom code to solve the business problem. This will massively reduce the supply chain problem.

Then have LLMs dedicated to scanning codebases for security issues.

Talking about the new normal is the new normal.