Remix.run Logo
▲ pron a day ago

> I said that first, and of LLMs being already, today, more persuasive. You inverted the implication.

I did not. Think again.

> Also, careful what stones you cast, you're in a glass house with a lot of the logical flaws of the stuff you've said already.

Possible, but I don't think so. Logic is my field (https://pron.github.io)

> False. Most leaders are above average intelligence.

Doesn't contradict what I said at all.

> Why did you bring it up then?

Because the question isn't "what can something in power do?" but whether intelligence necessarily lead to power.

> When you're already saying in your first post "This entire analysis is predicated on the assumption that intelligence confers some unlimited power,"?

I don't think you're even trying to understand what I write.

> LLMs and other AI already demonstrated the power of persuasion to an above average level.

Again, not relevant.

> Name any combination of diseases or insects you like that has the potential to wipe out the human species.

There could be microbes with the power to wipe out humanity (maybe not to the last human, but enough to destroy civilisation): https://pmc.ncbi.nlm.nih.gov/articles/PMC7122301/

Note that most papers are interested in likelihood, but my point is about potential. Microbes control more resources (and have more "computational capacity" at the biological level) than humans, and so they have more power even without intelligence.

Anyway, the general point is that it's resources, not intelligence, that confer power. Intelligence can compensate, but up to a point. And there is no direct correlation between intelligence and control of resources. Even among humans, collaboration in the use of resources among people often results in much more power than that of a much smaller group of more intelligent people (and BTW, plants also have more power because of their huge biomass (not claiming that they're likely to wipe us out)). The more concrete point is that a machine that is more intelligent than any human will not necessarily have more power than every human. "ASI", "aligned" or otherwise, does not necessarily lead to control. We could, of course, choose to give it control, but that's not a matter of technology, and there's clearly no social or historical reason to believe that that choice is inevitable or even the most likely (we don't choose to put the most intelligent among us in control).

My simple example was an "ASI" that tells us we must sacrifice something to control climate change, which happens to be what the most knowledgeable among us are already saying. Not to mention, of course, that different instances of such an AI are likely to disagree with each other quite a lot (because intelligence is insufficient to predict, control, or even fully observe nonlinear systems). Even partial success requires a political allocation of resources. Suppose that a clever AI says that it will be able to construct a more fair economy, but to do so it needs observations that require people to share more personal information and building that infrastructure requires raising taxes. The road from AI to machine overlords is not a simple one.