I think the even more important observation here is that the cat is already out of the bag on this approach to vulnerability research; it can be done effectively with open-weight models today.

I see it's an arms race between white and black hats using this new tool to expose vast new areas of vulnerabilities and fixing them to improve security as fast as possible. It's always been a cat-and-mouse chase, or maybe more like a whack-a-mole, but now accelerated with super-human ability on both sides.

Fast-forward a few years of this, I wonder whether they will have found all the low-hanging fruits (from AI perspective) and the situation will settle down; or it will keep getting worse - with AI development never reaching a plateau - until we admit that all existing software and hardware are hopelessly vulnerable and likely compromised already.