AI Superintelligence doesn't scare me for the same reasons "grey goo" doesn't scare me.

We are awash in self-replicating machines. The biosphere is already a grey-goo apocalypse. Any new competitors have a serious moat to cross to out compete any existing self-replicators.

We are awash in intelligent agents. Our society (and meta society) is full of superhuman agents already. There is a huge moat for any new intelligence paradigm to cross.

What I am afraid of is the existing superhuman agents (companies, governments and religons) will produce AGI or superintelligence and then proceed to use it as cognitive mitocondria, even further deepening thier supremacy in the cognitive ecosystem.

"Intelligent agents" we have around run on a metabolic budget of 25W and a hardware platform the size of a melon.

Human intelligence doesn't scale upwards well. Individual humans only get this smart, and there are gains from getting multiple humans to work together - but the more of them you add, the larger is your communication and coordination overhead. In no small part because humans are self-interested agents that simply aren't designed to compose their capabilities seamlessly. You can't get a vastly superhuman intelligence simply by piling together more humans.

Human intelligence doesn't scale sideways well either. Unskilled labor is cheap and plentiful, but if you have a human with a very specific skill, the process of getting more of that capability is very long and very involved. Often, it's easier to redesign an entire process to run on worse humans than it is to train more humans for better performance.

Institutions are more capable than individuals, but far less capable than the sum of individuals within them. At many corporations, the majority of individual productivity is absorbed by management overhead and corporate rot.

AI isn't bounded by those limitations.

AI can scale intensively and extensively. AI can be scaled up by upping the compute budgets. AI can be replicated and copied indefinitely. AI doesn't have the innate human "I don't live to work, I work to live" overhead. AI can outclass human intelligence by a long shot.

The "moat" that's there is already being eroded by modern day LLMs. Betting that future AI systems can't cross it is folly.

I have the same feeling. I'm not worried about superintelligent AI because we are only training them on human level intelligence. By what mechanism does our current AI technology take the leap to technologies that humans have never conceived of?

Our current AI is more like a fancy Google search than some kind of machine God.

I'm old enough to remember when grey goo and nanotechnology was the apocalyptic scenario du jour for a short time after some guy at MIT wrote a book, and because he was at MIT people took it seriously even though it was ridiculous. If someone at the University of Kentucky or Kansas had written such a book, it would have been ignored. When prestige manages to align with bad ideas, it's pretty awful, and it can derail the entire civilization for a while.

I was like... nanotechnology and grey goo already exist. It's called biology. The scenarios I was reading were silly. They violated conservation laws and laws of physics. But people were believing it and calling for limits on nanotechnology research.

I remember arguing with smart people on this, and that was when I started to realize that there's two kinds of dumb. I had the same realization later when I argued with an incredibly intelligent guy who was absolutely convinced the moon landings didn't happen. See, there's dumb-dumb and smart-dumb, and the people who thought grey goo would eat Earth or that the Apollo landings were a hoax were the latter. Smart-dumb is high-IQ rationalization of ultimately irrational and absurd ideas, and the smarter you are the more effectively you can do this.

I've met some really shockingly brilliant fools over the years who believe in all kinds of outlandish conspiracy theories, absolute literalist religious fundamentalism, idiotic political doctrines that directly contradict basic logic and all of lived human history, and so on. All of them can engage in sophisticated airtight rationalizations.

I sometimes wonder if this is one of the evolutionary forces constraining intelligence. In my experience, smarter people are somewhat more likely to believe highly sophisticated and complex stupid things, and they are much better at convincing others of these things. That's probably more dangerous to them, their family and friends, and the species than dumb people believing simple silly things that are easily debunked.

On AI...

Is AI potentially dangerous? Very. It's already dangerous in a number of ways. The biggest right now is probably mass production of personalized propaganda, mass surveillance, and mass manipulation. There's also the potential that bad actors could use it to accelerate their ability to make things like garage WMDs (biotech, chemical weapons, etc.). None of this requires hard take-off superintelligence. It's just inherent risks to a powerful technology.

These are not entirely new risks. They were already present in the Internet and computing. AI just raises them to a higher level.

The extreme hard take-off stuff is silly, and it actually distracts us from talking about the much more realistic dangers and coming up with reasonable solutions that don't also throw away the huge benefits of these technologies.