The cool thing about LLMs is that once a capability is "good enough" you can always "chain" them together for better overall results. On the client side this means "write an API that does x y z" -> "analyse this API for security concerns" -> "PoC for each finding from this report" -> "fix this code according to these verified claims".

On the "server side" (i.e. training) you can use the current gen models to improve the training data by running many parallel environments with a similar loop as above. Then incorporate the new data and repeat. Reminiscent of the old GAN approach, where the generator and discriminator are trained together in an adversarial regime. The end result should be safer code on "vanilla" prompts. "Write an API that does x y z" should now contain the learnings from this loop, and the models should produce better code.

Works really well for every verifiable scenario. And as the models become better, they can also more reliably create environments that closely match real-world scenarios. If you also have some data from human devs (say you run a subsidised coding model for a few months), even better.

An example of turning a "normal" repo into a verifiable environment that I read recently in the Cursor blog: take a repo, ask an LLM to remove a feature, verify that the app still works w/o the feature, verify that the tests for that feature fail. Ask a generator to "add feature x". Verify with the original tests. If pass -> give carrot :)

The key is composition. Once you unlock a new capability, that gets implemented and incorporated into the next training run. Pretty neat, I would say, and the main driver for the recent increase in the breadth of capabilities for new models.

> If pass -> give carrot :)

More like, give $$$ pass or not.