Its not entirely a prompt issue, but also what it was trained on.

Nowadays, the frontier is pretty decent all around, but it used to be generated php/javascript was questionable, ts was a bit better, and Rust was consistently decent.

Even just setting the max width of your README to 80 chars vs using full markdown could get you very different C code.

My take is that its pretty easy to write _much_ more secure code; use fewer dependencies and the ones you have should only do what you need them to do with no superfluous unused code compiled in. Configuring that used to be too much work, now its a single prompt.

Most software is written without giving a thought to security. Because security is a source of cost. That was the case before LLMs, I don't think it will change with LLMs.

Now I wonder: given that the LLMs are steered by the companies who own them, doesn't it mean that they can somehow make security better? It cuts both ways obviously: that a company can read and generate a large part of the code in the world is a risk.

As an open source maintainer, I can most definitely tell you that the quality of the average PR is now better. The problem I am facing is more that some people tend to "flood" and misuse AI in different ways. But their AI writes better code than they do.

I think that even worse is that AI might choose just to write something, and skip using a library that is well maintained

it costs time until the new Opus7 is trained on hardened code.

Depends where it is blind at.

Adversarial fixing. I like it

That study is from 2022, with at most a last update 2023? (if I read it right)

I'm not at all saying that "oh 3 weeks ago, discard", but three years ago absolutely has a "before and after".

Three years ago AI generated code could basically not be used at all, and I had to rewrite just about every line. Today I have to watch out for generated code making steps in the wrong strategic and architectural direction, and over engineering, no longer hallucinating nonexisting APIs and forgetting to check the auth token.

I still look for those kinds of mistakes, but I think citing a 2022 paper for "can AI do X" is not useful.

It just means that the safeguard we put in place over decades of software engineering with humans also need to remain in place when engineering with agents.

Having properly validated libs with separate concerns and a lot of eyeballs (and agent sensors) on the code so that it can be integrated cleanly in other products. Static code analysis for our apps. System prompts that require output to be secure, etc.

All of these practices that have been touted to be obsolete because agents can now just generate tailor made code just when and where you need, are anything but obsolete. On the contrary, they are paramount to the future of software engineering.

from your paper:

> Overall, we find that participants who had access to an AI assistant based on OpenAI's codex-davinci-002 model wrote significantly less secure code than those without access.

That model came out in March 2022, it's GPT-3.5 !!!

I'm sorry but it's absurd to use that as evidence for anything related to AI coding today.

You can't just hedge by saying "it's somewhat older". It's like discussing dunno a new train connection between cities and citing a paper measure steam locomotives.