I watched a video from him recently where he said open source models would win people's trust when it comes to AI. But I struggle to reconcile this with the above picture; if open source models become the norm, I feel P(Doom) probably will go up rather than down. At least the frontier AI firms have some guardrails in place (although after the Hugging Face hack they clearly aren't great). a wild west of everyone running their own model is surely worse in that regard.

A bunch of sites getting hacked isn't an existential threat. An existential threat would be something that could viably imperil the survival of civilization. The problem with existential threat concepts is that it generally requires fields where the limiting factor isn't knowledge. If somebody has the intelligence, resources, and desire to e.g. try to build a deadly biological virus, it's certainly not the lack of knowledge that's stopping them.

In general if we go back 50 years ago, when a single common phone today would have been seen as a super-computing device of unimaginable power, governments would likely have viewed it as a potential world-ending technology. It obviously wasn't and isn't. The conservatism would come solely from the fact of it being a revolutionary leap. That's all we're seeing happen with LLMs as well. In 30 years everybody will have vastly more competent AI systems running locally on whatever sort of digital devices we're using then, and they'll look back at with bemusement about us freaking out about people being able to query computers in natural language.

P(Doom) is an invented concept, relative to ones own version of "doom", can we maybe actually act like adults and not man children who are engaged in performative speculative fiction. There are real problems like inequality that P(Doom) loves to conveniently forget.