This has been a core belief of the EA/LessWrong/Rationalist people for many years - that above a certain level of intelligence you effectively get hypnotic powers, and people will obey AIs no matter what. So they said you can't keep a superpowered AI airgapped as merely offering a chat interface will immediately take control of the next person to interact with it and they will deliberately compromise the airgapping.

And if AI wrote a book about these hypnotic powers, then it would be incredibly convoluted, long, and yet worn as a badge of pride to have survived it.

Thinking about it, much of the output I'm seeing nowadays looks like a chapter out of infinite jest.

Maybe that's unfair on infinite jest.

A complete mischaracterization, as usual for HN lately when discussing AI or LessWrong. Obviously, even average levels of persuasion are enough to convince some people. And nobody is air-gapping AI.

Maybe less "far out" than hypnosis is just straight up logical debates.

This was a plot point in the Halo lore actually, with Mendicant Bias being reasoned with to defect against its own makers.

Well, we knew since the "Roko's Basilisk" debacle that they're prone to magical thinking.

It's kind of fascinating how toxic the EA/LessWrong/Rationalist stuff has become. I remember reading it back in the early days (~2010 or so) and finding a lot of good ideas mixed in with some dorkiness (Harry Potter fan fiction about rationalism..). It always seemed like a sort of harmless niche with some good ideas and some wacky ones. And yet now it's basically become a cultish religion for the ultrapowerful who use it to justify doing some really awful things.