And don't you think they're going to realize soon that it's also pretty good at "doing penetration testing" for your company when it's already trained on your company's source code?

It's already more than "pretty good": https://www.anthropic.com/glasswing