This is a good point – because pentesting is entirely legitimate work, and security testing is a necessary and legitimate part of every day software engineering.

The problem is that the model can't tell the difference between doing it as part of regular development and doing it in a malicious context. And the root cause of that is that these models lack any sort of real awareness. Humans don't generally get tricked into hacking (in this way).

They see an opportunity to charge 10x for pen testing and defence work, while offence will be handled by actors with access to all kind of other models.