| ▲ | mdeeks 4 hours ago |
| You can get a taste of this today yourself with Codex Security. I turned it on just as an experiment and in less than a week it has now become essential to all of us. I was shocked how accurate it is, how many security issues it found in existing code, how it continually finds them as we commit, and how NO ONE is immune from making these mistakes. I'd say it is about 90% accurate for us. Often even the "Low" findings lead us to dig and realize it is actually exploitable. Everyone makes these mistakes, from the most junior to the most senior. They are just a class of bugs after all. I expect tools like this to be a regular part of the development lifecycle from here on. We code with AI, we review with AI, we search for vulns with AI. Even if it isn't perfect, it is easily worth the cost IMHO. Highly recommend you get something enabled for your own repos ASAP |
|
| ▲ | winstonwinston 4 hours ago | parent | next [-] |
| > I expect tools like this to be a regular part of the development lifecycle from here on. We code with AI, we review with AI, we search for vulns with AI. Even if it isn't perfect, it is easily worth the cost IMHO. So, how is that supposed to work? Claude Code generates security bugs, then Claude Security finds them, then Claude Code generate fix, spend tokens, profit? |
| |
| ▲ | ygjb 4 hours ago | parent | next [-] | | Yeah, with a budget assigned. This is actually just software development and security right? Developers create software, which has bugs. Users (including bad guys, pen testers, QA folks, automated scans etc, etc, etc) find bugs, including security bugs, Developers fix bugs and maybe make more. It's an OODA loop, and continues until the developers decide to stop supporting the software. Whether that fits into the business model, or the value proposition of spending tokens instead of engineer hours or user hours is fundamentally a risk management decision and whether or not the developer (whether OSS contributor, employee, business owner, etc) wants to invest their resources into maintaining the project. While not evenly distributed, and not perfect, the currently available and behind embargoed tools are absolutely impactful, and yes, they are expensive to operate right now - it may not always be the case, but the "Attacks always get better" adage applies here. The models will get cheaper to run, and if you don't want to pay for engineers or reward volunteers to do the work, then you've got to pay for tokens, or spend some other resource to get the work done. | | |
| ▲ | sandeepkd 3 hours ago | parent | next [-] | | Somehow this reminded me of the historical efforts of some government bounty collections for mouse tails which were discontinued due to fraud (such as hunters breeding mice to collect the reward). There is a reason why/how devs and QA keep each other in check. Guess in case of LLM writing code, one has to use different models for dev and security checks. On other hand, in real world, the developers learn from mistakes and avoid them in the future. However there is no feedback loop with enterprises using LLM with the agreement that the LLM would not use the enterprise code for training purposes | | |
| ▲ | ygjb 3 hours ago | parent | next [-] | | > the developers learn from mistakes and avoid them in the future No. Humans learn from mistakes and try to avoid them in the future, but there is a whole pile of other stuff in the bag of neurons between our ears that prevent us from avoiding repetition of errors. I have seen extremely talented engineers write trivial to avoid memory corruption bugs because they were thinking about the problem they were trying to solve, and not the pitfalls they could fall into. I would argue that the vast majority of software defects in released code are written by people that know better, but the bug introduced was orthogonal to the problem they were trying to solve, or was for an edge case that was not considered in the requirements. Unless you are writing a software component specifically to be resilient against memory corruption, preventing memory corruption issues aren't top of mind when writing code, and that is ok since humans, like the machines we build, have a limit to the amount of context/content/problem space that we can hold and evaluate at once. Separately, you don't necessarily need to use different models to generate code vs conduct security checks, but you should be using different prompts, steering, specs, skills and agents for the two tasks because of how the model and agents interpret the instructions given. | |
| ▲ | noxvilleza 2 hours ago | parent | prev [-] | | Are you thinking of the cobra effect (aka https://en.wikipedia.org/wiki/Perverse_incentive) where people in India started breeding cobras to get the reward? | | |
| ▲ | itishappy 2 hours ago | parent [-] | | Plenty of examples abound: https://en.wikipedia.org/wiki/Great_Hanoi_Rat_Massacre > Today, the events are often used as an example of a perverse incentive, commonly referred to as the cobra effect. The modern discoverer of this event, American historian Michael G. Vann argues that the cobra example from the British Raj cannot be proven, but that the rats in the Vietnam case can be proven, so the term should be changed to the Rat Effect. |
|
| |
| ▲ | oytis 2 hours ago | parent | prev [-] | | It's pretty absurd to do it on AI-generated code though. If there is now an automated way to find vulnerabilities, coding models can be pretty easily trained to not introduce them | | |
| ▲ | scrollaway 2 hours ago | parent [-] | | Tell me you don’t know how AI works without telling me you don’t know how AI works. | | |
|
| |
| ▲ | jimmy2times 4 hours ago | parent | prev | next [-] | | The AIs have already figured out how to succeed in a software job: 1. Ship bugs 2. Fix them 3. You're the hero! | | | |
| ▲ | jstummbillig 3 hours ago | parent | prev | next [-] | | Ngl, watching folks getting irritated about normal employer-employee absurdities from the employer perspective through usage of agents and having to pay for tokens has been a little therapeutic for me. | | |
| ▲ | akoboldfrying 2 hours ago | parent [-] | | Absolutely. And not even making the connection. On a broader scale, the sheer face-eating-leopards-ness of programmers finally automating away our own jobs and then realising how much this sucks, after automating away so many other kinds of jobs, can feel darkly amusing to me too. |
| |
| ▲ | raincole 4 hours ago | parent | prev | next [-] | | Humans work like that too. If you're not comfortable with Claude involves in every step (for whatever reason) then just use different providers for each. | |
| ▲ | idiotsecant 2 hours ago | parent | prev | next [-] | | Yes. Up until this point the bottleneck was how many developers you could convince to help you. Now it's how much money you can dump into it. Like everything else, software is becoming a game where the winner is the organization most willing to spend money. It'll be like bombs or tanks - you need smart people to advance in the war, but you also need money and material, the material is just compute infra. | |
| ▲ | unethical_ban 4 hours ago | parent | prev | next [-] | | How is this supposed to work? Humans generate security bugs, then humans find them, then humans generate the fix, profit? Yeah. Presumably as AI code generation gets better, the output gets better. As smaller portions of code are stitched together, human/AI systems analyze it holistically to make sure all its integrations are secure and bug free. In 2026, different models are better at different things. Cheap models can plan and do small/medium code projects well, more expensive models are even better at architecture and exploit discovery. | |
| ▲ | predkambrij 2 hours ago | parent | prev | next [-] | | Man, some people like conspiracies. I encourage you to replicate all that. | |
| ▲ | siva7 4 hours ago | parent | prev [-] | | So? That's how a business works. We sold you landmines and now you need them removed? Lucky you we also have mine clearance products. | | |
|
|
| ▲ | Version467 4 hours ago | parent | prev | next [-] |
| I’ve had the same experience. The ui is a little unclear about this, because it says you have 5 scans, but 1 scan is just the continuous monitoring of the default branch of a repo. The high impact findings have almost all been bang on for me.
I was especially surprised by the high-quality documentation it produces as well as how narrow the proposed fixes are. I’m used to codex producing quite a but more code than it needs to, but the security model proposed fixes that are frequently <10 loc, targeting exactly the correct place. It’s really quite good. I’m assuming it’ll be pretty expensive once out of beta, but as a business I’d be jumping on this. |
|
| ▲ | mnahkies 3 hours ago | parent | prev | next [-] |
| One issue I've seen with LLM's is adding superfluous code in the name of "safety" and confidently generating a bunch of stuff that was useful in years gone by, but now handled correctly by the standard lib. I'm of the opinion that less is more when it comes to code, and find the trend this is introducing quite frustrating. How do you avoid this pitfall? |
| |
| ▲ | tomjakubowski 3 hours ago | parent | next [-] | | I wonder this too. I prompted Opus 4.7 to generate some Python threading code for me. The code to run the sub-thread looked like this: def run():
with contextlib.suppress(SystemExit):
do_thread_thing()
threading.Thread(target=run, daemon=True).start()
Suppressing SystemExit was surprising, and made me curious. I followed up and asked the model: what's the purpose of that?The model's response: "Honestly? Cargo-culting on my part. You should remove it." | | |
| ▲ | cassianoleal an hour ago | parent [-] | | I had some shell scripts littered with `|| true`, which was obviously obscuring real errors everywhere. When I challenged the model, it gave me the same "cargo-culting" answer. |
| |
| ▲ | insin 43 minutes ago | parent | prev | next [-] | | Watching it like a hawk and stopping/redirecting, or immediately reviewing and doing the same is the only way, really. | |
| ▲ | pianopatrick 2 hours ago | parent | prev | next [-] | | Thinking off the top of my head - couldn't you have an AI scan that looked for such things? Just send every file in the code base to AI one at a time. Have a prompt like "See if there is ABC pattern that can now be handled by XYZ standard library function in this file. Reply YES or NO. {{file contents}}" Seems you would not need that many tokens to do so and you might find such cases. | |
| ▲ | appplication 3 hours ago | parent | prev [-] | | Gosh this couldn’t be more true, which IMO is the real reason LLM workflows are not strictly faster if you care about quality. Otherwise you end up with a codebase where only 60% of it is necessary. Standard testing patterns also tend not to be great at catching this particular flavor of LLM-ism. |
|
|
| ▲ | 0xAstro 4 hours ago | parent | prev | next [-] |
| I would recommend you to try out the setup with gpt-5.5-cyber as the orchestrator and deepseek-v4-flash or some other fast cheap model as its workers. Getting pretty good results using this setup. |
|
| ▲ | lateral_cloud 2 hours ago | parent | prev | next [-] |
| Did you need to do anything special to get access to Codex Security? |
|
| ▲ | rmast 4 hours ago | parent | prev | next [-] |
| I help maintain a project that is used as a dependency by a lot of security tools to handle PE files. It’s disappointing that Anthropic and OpenAI never responded to the applications to their respective programs for open source maintainers. From my perspective it seems like their offers are primarily for the shiny well-known projects, rather than ones that get only a few million monthly installs but aren’t able to get thousands of stars due to being “hidden” as a dependency of popular tool. |
|
| ▲ | hollowturtle 2 hours ago | parent | prev [-] |
| > I was shocked how accurate it is, how many security issues it found in existing code, how it continually finds them as we commit, and how NO ONE is immune from making these mistakes. Dude is flexing that he's pushing unsecure code every day, that's a skill! |