Just gave this a try, pretty interesting how a simple python script generated with two un-named models uses requests library version with CVE's. The scary part is, the script ran. This changes things in terms of leveraging AI. I will come back with more feedback soon, but for now, this is amazing

▲

jaimefjorge 5 days ago | parent [-]

Hey thanks for testing! That's been my experience well, it's very frequent to see libraries with vulnerable versions being introduced in code. What's also interesting is that, despite using incredible AI coding models like Sonnet 4, you still get CVEs in your code. Try this with Codacy Guardrails: "create a Java server using undertow".

Thanks for testing. Please do share your feedback when you test further!

▲

im3w1l 4 days ago | parent [-]

I mean it's almost inherent to LLM's right? Like they only know about version before it's knowledge cutoff. I guess it's a big argument for not putting exact versions in files generated by LLM, only major (+minor?)

	▲	jaimefjorge 4 days ago \| parent [-]
		Yes. My point is that because of training cutoffs it should be mandatory to run SCA scans when dealing with AI code generation. Not putting exact versions would be a good idea. But that’s not what’s happening today.