I am finding it hard to read these deeply impassioned letters while keeping in mind that they are spending millions to train models at scale to do the exact thing they say they are worried about them doing. Not general "intelligence" or "reasoning" but fast and effective offense.

Like why are you explicitly RL-ing your models on exploit generation, scoring them on a public benchmark called ExploitGym, if you have specific concerns that rogue models will cause "cyber incidents"? Sure you can check the capability, you can teach offense to learn defense, but it seems like they are literally benchmaxxing it. Why?

OpenAI are like, oh no, while competing in our "advanced PhD level cheating techniques course" our models unexpectedly cheated in a way that we absolutely could not have foreseen. "We need to slow down. Somebody please stop us". The thing that is unaligned here is not the models. Everyone in the story (especially the humans) just keeps doing what they think will get the most reward.

They are writing these letters to investors. You can read them all as: Our models are so good it’s scary.

Million? BILLIONS. So I ask myself: are they calling for a pause for the sake of mankind? Or are they proposing to put a throttle on their own gargantuan investment plan ?