Whenever I hear some big AI CEO say that AGI/ASI might end humanity, I imagine it doing so by conditioning people to commit suicide or some kind of massive-scale manipulation. Probably setting people up into complicated bad situations and then blackmailing them.

I don't know if this is it here. These sorts of jobs can have tough moral implications. But it seems like a hierarchy of blackmail has been set up in our society and it seems like a dangerous vulnerability.