This argument should be reframed as "AI has the potential to cause catastrophic loss of life". It's too high of a bar to prove that AI will "wipe out humanity". It's sorta like climate change, it's easier to show that weather patterns are becoming increasingly dangerous.

Why bother with killing, when it is much more fun to just make everyone's life miserable instead? ;)

> While Fraser's explanation was technical and specific, he hit a light-hearted note to summarise it: "LLMs [large language models] won't wipe out humanity because they just don't have that dog in them."

Never underestimate the drive of a good while loop.

>Coxon has said a group of AI agents, based on AI models that do not currently exist, could decide to create and then aim a biological weapon, but he did not detail how exactly that would take place.

I can't imagine why someone would be reluctant to outline details of how to build a bioweapon to an audience of millions.

At a business dinner this week with 3 high level eng clients p(doom) in next 10 years was brought up. It was rated by 2 if them them at 1% and 30% respectively, with 3rd guy demurring to put a number but saying its > 0. Completely unprompted, from people whose company is very AI-all-in.

Will AI become conscious and kill everyone?

Or will the conscious beings that exist today use it to kill everyone?

I don't think it actually needs to be conscious, or understand legal or moral issues.

Just tell it everything is a simulation and it is completely sandboxed from harming anyone.

Anyone who claims AI is going to kill us I just assume is either an idiot, not knowledgeable enough about the technology to have a worthwhile opinion on it, or is fearmongering to hype their AI investment.