> what separates it from religion?

The rational methods used to establish a large P(doom).

To definitively falsify the claim, humanity would need to live alongside superintelligence forever without it wiping us out. That doesn't mean we can't be pretty darn sure though, in the same way that we can be pretty darn sure that nuclear war could end civilization.

Those "rational methods" are essentially just pascal's wager. You stick some ~infinitely bad outcome (nasal daemons torturing simulations of you and everyone you love for all time) and/or ~infinitely good outcome (perfect paradise for as many sentient minds as could fit in the physical universe), then you can get whatever conclusion you want by multiplying any finite probability by these things, and even give yourself bonus asspats for being conservative by adding several order of magnitude fudge factors against your claims.

The rationalist doom and sex cult uses convoluted mathematics inspired language to justify their faith-- but at the end of the day it's just faith no matter how they dress it up.