> The rate at which we're churning out AI-generated code is going to make this worse

Not only this, a significant number of developers have stopped looking at code, trusting the bots. I've seen major security regressions, bot rolled auth implementations, and falsified benchmarks... just this last week, all in projects that get hyped.

Rule #2 - do not blindly trust Ai output

https://susam.net/inverse-laws-of-robotics.html

▲

user43928 5 hours ago | parent [-]

It's not stupid if it works.

Concretely, I would like to see it benchmarked:

Have Opus 5.5 implement the authentication, with a 6.1 Sol reviewer. Out of 100 times, how often is the result insecure?

I wouldn't be surprised if it's zero times.

There is also the possibility of specifically prompting for a security audit on top, which would perhaps further lower the risk of overlooked issues.

Now one can say that due to the non-deterministic nature of the tools, there is always a non-zero chance of producing insecure implementation.

That's not really important here - obviously manual review isn't deterministic either, the risk will never be zero.

▲

verdverm 5 hours ago | parent [-]

The way the bots fundamentally work means that we can never trust their output. We have ample cases at work where bots catch human mistakes and humans catch bot mistakes. The important aspect here is not completely handing things over to the bots.

> I wouldn't be surprised if it's zero times.

Statements like this concern me.

▲

user43928 5 hours ago | parent [-]

If you're so concerned you can create such a benchmark and show us all how wrong my guess is.

Instead I mostly read such opinions based on active imagination and no concrete experience with using AI for such work in a setup as described.

▲

verdverm 4 hours ago | parent [-]

I suspect you would disregard or find fault in any benchmark I or others provide you.

Alternatively, if you want to see a benchmark so badly, you can create such a benchmark for your guess. I would imagine you would have more faith in the results of your own work.

▲

user43928 4 hours ago | parent [-]

If someone were to genuinely tell me that with the two SOTA models with the review step, severe security issues were left in the code, I would believe it.

That's not what anyone tells me though. I only hear theorizing and vague references to how AI failed people.

You're spot on that the thing I trust most is my own experience. I'm mostly working on mobile apps, my project does not have user accounts, so I couldn't tell you about authentication.

However, it is my experience that these models got very good at catching high-priority issues if you use the review step as I described.

Thus I theorize that SOTA models are likely to produce secure implementations.

▲

verdverm 3 hours ago | parent [-]

"secure implementation" is vague, for example, here is an OpenCode issue (links to related PRs/follow ups) that I worked with their team through to improve security.

Our hunch is that the Ai told them this was ok to do, but it broke invariants users relied on. Perhaps it was not "hackable" in the sense I think you are getting at, but it did open a very big security hole through choices and design. The critical thinking was lacking (imo).

https://github.com/anomalyco/opencode/pull/51337

The bots make mistakes all the time, even through multiple rounds of reviews. Thus I have zero faith that they will produce secure code reliably 100% of the time. More Ai does not magically fix this, they go down the wrong path and double down, or work around incorrect assumptions.

This is a good talk by a member of the Linux security team: https://www.youtube.com/watch?v=NnV_cWeoo5Q

Also worth noting that OpenCode Console was hacked days after the configuration regression and two PRs to change it back. Notable because it shows a pattern.

https://news.ycombinator.com/item?id=50133401

	▲	user43928 2 hours ago \| parent [-]
		A poor design decision in OpenCode configuration handling is interesting, but not directly relevant to the question. Assuming the question is whether not reviewing generated code is likely to create vulnerabilities despite SOTA models having audited the implementation. Here it seems to have been a design choice, rather than an implementation mistake. I do not know what models were used and whether a review or audit has been done. We are basically back at a vague the AI failed. And sure, this happens, but there are some strong claims I'm reading here that are imo not evident: - not reviewing implementation is foolish - doing so will result in a flood of insecure software - more AI cannot fix this -- My arguments on each point: - depends on risk tolerance, attack surface, error rate - depends on actual performance of the proper setup I described on typical tasks, compared to the average engineer - more AI can fix this, provided the audits become accurate enough to identify vulnerabilities with eg. 99.9% reliability -- Are you sure that such accuracy for auditing is impossible, and will remain impossible in the future?