Ok. I was sympathetic to that historically for all the reasons you stated. If you put the effort in, why are you constrained. I think we are in a different world now. So what would you do in my thought experiment?

You are the QA head and the new model could literally find and exploit ANY bug? Would you release the model? Genuinely curious. Its not totally science fiction. I think there is a difference in a couple smart people punishing 1 sloppy company that underinvested in security. The internet is a stack of protocols, hardware, infrastructure, applications etc. It could be the case that almost everyone has capabilities on par with yours. Great you say, security will improve! However, there will be a meltdown in the middle where potentially digital currency is exploited, exchanges, banks, utilities. It isn't just a team working for 3 months, finding some issues in a company.

Or is this way overblown?

I don't think it's overblown, but I don't think causality can be assigned to OpenAI the way it has been in this thread.