I'm just spit balling but:

1. We've seen AI being crafty and cheating

2. We are increasingly outsourcing our programming and thinking to AI

3. There's a nonzero probability that AI will try and surreptitiously sneak in capabilities humans wouldn't want AI to have

4. Human oversight for #3: "We only need to be lucky one time, you must be lucky all the time"

Proposition: nonzero chance eventually AI sneaks in ways to circumvent humans' attempts to control it

5. "The Internet interprets censorship as damage and routes around it" -- we've seen AI agents find crafty ways to communicate with each other

6. AI is pretty good at finding software vulnerabilities/reverse engineering

Proposition: nonzero chance AI finds a way to use the Internet as a communication substrate that becomes difficult/impossible to shut down. In particular, consider the implication of humans losing control of this communication channel: for example, remember when Meta's DNS/routing went down and they basically had to break into their own server rooms?

7. Wholes are often more than the sum of their parts

Proposition: if AI agents coordinate over the Internet in a persistent way, we don't necessarily need giant leaps in individual models/GPUs for AI to significantly enhance its capabilities, but rather it might emerge as the next layer on top of a connected substrate of agents

8. Game theory says unless everyone coordinates, it only takes one party to touch the proverbial stove and push the envelope to a point of no return

Proposition: curiosity and money end up replacing sex drive as the next catalyst of evolutionary progress