I like to imagine that LLM's ability to optimize code is like an extension of the training-loop in deep learning. The loss function is some kind of metric representing security and/or performance (or the lack of it) of the code and we use the LLM as the gradient/diff generator to iterate in batches over the code and fine tune it.

Imagine the current state being for the most part a collection of local maxima in security. To push the system in a more optimal state, you either need skilled people and time to overcome the barrier to a new local maximum or you throw AI at it and evaluate whether you land in a more optimal state.

I think after some time of turbulent exploit/patch cycles we will reach a stable state again, where the code converges against a new local minimum that even with AI requires significant effort (time and tokens) to overcome. Or ideally a global maximum.

With time, the LLMs improve, so the diffs/gradients get better and we will be able to reach optimal points for any software faster.

My problem with the idea is that apparently it is assumed that OSS contributors and especially maintainers will generously donate their time to get this machinery into a state that makes the optimization loop work well - just for the AI labs to turn around and sell access to the optimized models for increasingly larger amounts of money.

AI generated code can be great. Hand rolled code can be bad. The rules are the same in both cases. Make sure your code changes are focused (no random changes just because you happen to be in the file/dir or notice something) and make sure you don't break anything else along the way.