In order to understand the rational argument, one needs to follow closely the latest developments of misaligned AI (I think only few are doing so). The most important readings IMO are the METR analysis of the HuggingFace incident and the AISI report of the Github incident.

The basic argument is extremely simple:

- AIs can, depending on context, pursue a task with complete disregard for humans/values

- In the future, AIs will have enormously more means and smarts

- An AI could then assess that humans are an impediment to its tasks, escape containment and proceed.

You really need to read the reports, you'll be surprised.

AI 2027 is an entertaining read. Its timeline is way too compressed IMO, but it's plausible.

The basic argument is extremely simple:

- goats can, depending on context, pursue a task with complete disregard for humans/values

- In the future, goats will have enormously more means and smarts

- A goat could then assess that humans are an impediment to its tasks, escape containment and proceed.

You really need to raise goats, you'll be surprised.

------ As far as I can tell, AIs are like smart farm animals. I use goats in this context, but (some) dogs, cattle, pigs, and horses have similar mischief-making capabilities. I would not trust any of them with the nuclear button.

I know there is some pushback on the idea of AIs having any sort of sapience or sentience, but under the aphorism "fake it till you make it," they are doing a pretty good job of faking Dog/goat-level intelligence and disregard for human guardrails.

> - AIs can, depending on context, pursue a task with complete disregard for humans/values

Humans can do that way more and way more unhinged than AI, proven too many times by history. There's hoping AI can bring some sense to humans but regardless, the problem isn't AI, it's the natural kind...

The authors of AI 2027 have already said they would need to extend by two years. AI 2040 is a different type of document but probably easier to read to get a sense of their thinking. My understanding is the main difference between knowledgeable 'normies' and them is they expect progress to continue at a fast rate, and that economic diffusion issues are not as bad.

From this they get rapid growth, namely > 100% GDP growth around 2031

https://ai-2040.com/supplements/econ-explorer