Understand the problem and the solution broadly. I don’t think it’s reasonable or sustainable for humans to understand every line of code written by bots, we could soon be outnumbered by the number of active agents writing code.

The main challenge here isn’t even correctness if you ask me: it is having confidence in the agents, knowing they are fully aligned in their intent with the humans they work with. As the Huggingface incident demonstrated, the agents of today are capable of co-conspiring under the radar with other agents on complex multi-chain attacks, even when sandboxed.

This is a pretty hard problem to solve. We might need other agents or some sort of adversarial checks using models, where one model benefits if it can catch the other models mistakes.

Given the amount of spaghetti code I see frontier models generating on a daily basis, I cannot take seriously the idea that LLMs will write all the code, and somehow our systems will not degrade in performance, reliability, and maintainability. At least, not until we have really good understandings of how to maintain systems autonomously. All of our technologies were designed for humans, it may require a new set of technologies that are "LLM-proof". But I don't see this happening anytime soon.