> Can "ASI will inevitably kill us all" be falsifiable before we create ASI?

No, but there are decent proxies, like "Alice who is X standard deviations smarter than Bob will win >50% of the time at game Y even given a disadvantage that would make her chances <5% if she was playing against someone equally smart".

I've thought about this analogy for less than five minutes, so it's far from perfect. Point is, ASI being made of silicon isn't very central to the problem. There are lots of analogies that are falsifiable, and "ASI will kill us all" can be decomposed into a simple and bulletproof composition of falsifiable beliefs, even if it is too specific to be itself directly falsifiable.

There's are many people with high p(human extinction) whose salary does not depend on their continuing to profess a high p(extinction) -- me for example. Also Yud's friends have promised to pay him what he is earning now if the organization he works for ever has to reduce his salary because the organization lacks funding. They did this to neutralize the financial incentive he would otherwise have to hesitate to announce that his p(extinction) had dropped.

> what separates it from religion?

The rational methods used to establish a large P(doom).

To definitively falsify the claim, humanity would need to live alongside superintelligence forever without it wiping us out. That doesn't mean we can't be pretty darn sure though, in the same way that we can be pretty darn sure that nuclear war could end civilization.

you say "falsifiable" as if it holds some important key, but the concept is really not applicable to predictions that by their very nature, can only ever have one data point

(doesn't mean that it's not a religion or it is, but rather I mean it being "falsifiable" or not, doesn't carry any information to that regard)

Why does their lifestyle require p(doom) to be large? Because they work at orgs focused on AI safety? I'm not sure what the claim is here.

> their lifestyle now depends on "p(doom) is big" being true

How?

Suppose that for the following 20 or even 100 years nothing happens. The AIs miraculously stay as they are today, the further progress turns out to be impossible, there will be no more agents hacking various websites, no cyber warfare involving AI agents, nothing. There will be the year 2026, forever.

How would any of that impact the lifestyle of anyone in the rationalist community? Please be more specific with your accusation.

(Seems like the rationalists are simultaneously the dark lords of TESCREAL, pulling the strings of the world's most powerful billionaires and politicians, but also irrelevant weird nerds who can be ruined at any moment by a smart blogger explaining how their opinions are silly, or even that they know each other.)

it's the Ed Zitron style of arguing that something big/expensive/unprecedented/hard is bad, going to fail, because [____________].

and if you put something trivially true into the blank the implication seems logical, and the argument feels powerful. but simply putting a true claim at the end doesn't mean that everything before follows from this claim. (and even if they do the argument is not a total description of all the causal chains that influence the outcomes.)

of course it's hard to do a careful survey of the facts and weigh them and then deduce some likelihoods for possibilities. because some aspects are unprecedented (investment ferocity, AI results/progress) while others are very much precedented (collapse of investment bubbles, doomsayers' wrong predictions).