> superintelligence is, like rocket fuel, an extremely powerful force that has a default tendency to break containment and go boom

what evidence do we have this is the case?

We can learn from history: Albert Einstein famously tricked humanity into building nuclear weapons for him and was only prevented from wiping out all sentient life by the Princeton IAS Board of Alignment who published a very compelling blog post about realigning the A-bomb contra paperclips.