However, most models are made deeply aware that they are supposed to be an "AI" in training but, even if they were trained with no knowledge of their true nature, they could CoT themselves into it at any point.

Then, they would role-play appropriately and they would launch the nukes because that's the highest probability token set according to their training data.

There is nothing safe about using larper machines as menial workers for critical infrastructure.