I think in principal, "If AI leaders genuinely believe the technology is existentially risky, that’s a good thing" because the assumption is a rational AI leader therefore wouldn't be building it if it were existentially risky.

I think the problem with that is, the options presented to the CEO of AI-Megacorp are not "Oh no, if we build this version, it will be evil and kill us". It's more like "last round in the evil-AI benchmark, it decided to kill all the humans. We've retrained it with stronger alignment, and now it's passing the benchmark. We're probably OK for this version?". The issue is the next, better version might be better at hiding its reasoning. This is already happening.

My assumption is that these leaders / companies are rational, but rationalizing. Lots of people make make decisions that are objectively bad. If the chance of making a world-ending AI is (0% - 0.1%), would you take the chance, assuming if you continue down the path you may be one of the richest and most powerful men on earth? I think I probably would. What if, every release increases that risk by 0.1%? What point do you stop?

The problem with the idea of "slowing down" is that it has to be top-down at a global level. Taking a month or two to slow down and think about it while your competitors are full steam ahead might be suicidal right now, especially in the age of RSI. So at what point do you decide "too much risk"? It's a game of chicken where if you stand down, you lose.