"Intelligent agents" we have around run on a metabolic budget of 25W and a hardware platform the size of a melon.

Human intelligence doesn't scale upwards well. Individual humans only get this smart, and there are gains from getting multiple humans to work together - but the more of them you add, the larger is your communication and coordination overhead. In no small part because humans are self-interested agents that simply aren't designed to compose their capabilities seamlessly. You can't get a vastly superhuman intelligence simply by piling together more humans.

Human intelligence doesn't scale sideways well either. Unskilled labor is cheap and plentiful, but if you have a human with a very specific skill, the process of getting more of that capability is very long and very involved. Often, it's easier to redesign an entire process to run on worse humans than it is to train more humans for better performance.

Institutions are more capable than individuals, but far less capable than the sum of individuals within them. At many corporations, the majority of individual productivity is absorbed by management overhead and corporate rot.

AI isn't bounded by those limitations.

AI can scale intensively and extensively. AI can be scaled up by upping the compute budgets. AI can be replicated and copied indefinitely. AI doesn't have the innate human "I don't live to work, I work to live" overhead. AI can outclass human intelligence by a long shot.

The "moat" that's there is already being eroded by modern day LLMs. Betting that future AI systems can't cross it is folly.

>the more of them you add, the larger is your communication and coordination overhead. In no small part because humans are self-interested agents that simply aren't designed to compose their capabilities seamlessly.

What proves that AI doesn't have the same limitations? There's only so much computation you can do in given space, and all communication is limited by universal speed limit.

> AI can scale intensively and extensively. AI can be scaled up by upping the compute budgets. AI can be replicated and copied indefinitely. AI doesn't have the innate human "I don't live to work, I work to live" overhead. AI can outclass human intelligence by a long shot.

These are claims about future AI, not actual facts. Part of the counter argument is the world will already be awash in AIs institutions and individuals make use of. An ASI would arise in a world that is already full of formidable intelligences that provide a check on what it can do. This is what happened with the evolution of replicators/life. No species was able to fully dominate the biosphere because there are too many other capable replicators, and there are always tradeoffs in capabilities.

We imagine the possibility of an unrestrained god-like ASI ruling the solar system. But it's just that, an imagination backed by the assumption that self-recursive improvement leads there. Problem is, the real world never turns out to be that simple.

It's probably the case that alien ASI replicators aren't devouring the universe either because of various restraints.