The LLM didn't have a prod key. It found a prod key in the source base and used that instead of the key it was given.

The access is supposed to be managed in a way that prod would only be accessible with multi-user approval. And that's without even mentioning the fact that storing a key in the source code is a big no-no.

If an LLM can just do whatever after discovering a magic key (in the source code, of all places), with no multi-user approval, it is pretty much the poster child example of an issue with the process that I was talking about earlier.