Lots of people quibble about this and the details, but the general outline of the argument is simple and given enough time seemingly inevitable. I don’t see an easy way to attack this. Claims like “it’s not so bad”, “LLMs ain’t it”, “AI can be good” lack substance and come across as “copium”. This is fundamentally about intelligence itself and how we can control it. Not “hope for the best” or “give it our best shot”, but really nail it down like we nail down mathematical proofs. We got one shot at this, just the one. You blow it, it’s over. Ash and silence forever. Once you realize this it can be quite sobering.

IMO the only real counter argument I came across came from Joscha Bach which basically boils down to: we are doomed already, AI is literally the only realistic shot we have at outliving the next couple centuries, say. It’s also sobering but a but more optimistic.

> fundamentally about intelligence itself

Most of the concerns raised here are not about intelligence though. They're about Human intelligence.

We keep anthropomorphizing the machines, but the truth is the do not "want" the things we want and there is no reason they should, even in theory.

People just suck at imagining anything that might have different goals. See 'dark forest theory' for an example of the same logical flaw applied on a cosmic scale.