That stupidity is happening? Even after the Huggingface hack, frontier labs are using internal models to further their research. i.e. RSI is happening now and we're facilitating it.

To make the the argument that P(doom) is real and worth considering, you don't have to say that a fast takeoff is very likely. You don't have to make the argument that RSI to infinity is going to be super cheap, barely even an inconvenience. You just have to show that it has some non-zero probability. And then you start weighing probability of extinction versus finite, mild discomfort now. I don't think anyone is arguing we should let people starve to slow AI progress, instead just some capitalists make less money soon.

There are arguments against taking P(doom) seriously that lie in something like having exponential (instead of hyperbolic) time discounting of utility (so you can take the entire future of humanity as a finite utility value). Or in saying that P(doom) is zero or infinitesimal.

"Build it and Pray" is the default strategy that we're in, but it doesn't have to be the strategy we choose, and it's unlikely to be the best strategy.