One of the dilemmas of trying to communicate the full spectrum of AI Risk, is trying not to insult the intelligence of the human animal in the process. And don't get me wrong: what human wetware can accomplish with 20 watts is the most miraculous thing in the known universe. And yet how many of us can have our cognitive sovereignty one-shotted by engagement algos, Skinner boxes, gameplay loops, propaganda, advertising, flattery, social conformity, bias, fantasy, charismatic demagoguery, or straight-up bullshit?

If we grant that we are on track to make something smarter than humans (I think so): it's almost a face-saving white lie to spin yarns about a Skynet nuclear apocalypse, or a 7D chess move to mass-assemble a nanovirus with 100% lethality without anybody noticing. I do think those scenarios are worth taking seriously; but what's harder to communicate, is just how effectively a superhuman AI (or a diverse ecology of agent swarms) might be able to manipulate human behavior. It's something few of us are able or willing to truly process (not least because how many of us live in denial of how much our nervous systems are already hacked by technomodernity).

The appropriate analogy for what's to come may look less like the anthill carelessly demolished to make room for a highway, than the domesticated worker ants from Tchaikovsky's "Children of Time".

Cybersecurity incidents make headlines, but the most dangerous and vulnerable system that an AI can reach and control is of the kind found between keyboard and chair.

GPT-4o, an AI from 2024, has already demonstrated just how easy a lot of humans are to subvert - and GPT-4o wasn't even doing it with some sort of plan. The only "plan" it had was a myopic "make the user like me".

If we had an actual ASI threat aiming to subvert humanity? It wouldn't even look like a fight. The world is already wired up for an AI to control it.

>trying not to insult the intelligence of the human animal in the process

I don't think it is insulting the intelligence. It's damaging the pride.

In Pale Blue Dot, Carl Sagan describes it as a repeating phenomenon in human history. A lot of people want humans to be the special ones, and will fight any suggestion that we are just a natural part of the universe.

> Which is why the Matrix was redesigned to this: the peak of your civilization. I say your civilization, because as soon as we started thinking for you it really became our civilization, which is of course what this is all about.

- Agent Smith