We live in a world where many people, even in advanced countries, reject 18th- and 19th-century science, let alone more modern science. I don't understand why people will necessarily cooperate with any intelligence, human or machine, that tells them things they don't want to hear [1]. Not to mention that many if not most systems in the world are non-linear, and so no amount of intelligence can predict and control them, certainly not without gathering a lot of resources. Even the question of "alignment", which is presented as a simple matter, is obviously not so simple, because humans are far from being aligned with each other [2], and their values sometimes differ (to the point of coercion or killing).

This entire analysis is predicated on the assumption that intelligence confers some unlimited power, when both theory and observation [3] tell us that that is not the case. A super-intelligence is either unnecessary or insufficient for some of the "futures" here.

[1]: For example, an "ASI" will tell people that man-made global warming is real and they should probably sacrifice something to do something about it, which is also what human intelligence says. If that message comes from a machine, will people be more willing to accept it?

[2]: I don't think I'm "aligned" with Elon Musk or the US government, and they already have some power over me, not thanks to their intelligence but thanks to their control over resources.

[3]: Not only are the most intelligent people not the most powerful people, but there are biological species that are more capable of destroying humanity than the other way around, even though they are far less intelligent than humans.

Let's take the flipside of your hypothetical [1]: What if it actually tells them man-made global warming is real but insignificant compared to other forces outside of human control that are acting on global climate?

> I don't understand why people will necessarily cooperate with any intelligence, human or machine, that tells them things they don't want to hear [1]

LLMs are already more persuasive than most human writers. And LLMs are not the only form of AI: the algorithmic feeds which people now find addictive in the sense that they struggle to stop themselves from wasting time on them, got this way in part due to AI constantly optimising for just that.

Still, this is not universal. I am fortunate, because my mind is alien to the models these algorithms have of normal humans. Not autism, whatever oddity I am does not seem to be anything that anybody cared about enough to turn into diagnostic criteria, but I am definitely odd, and in this decade that's a good thing.

> This entire analysis is predicated on the assumption that intelligence alone confers some unlimited power, when both theory and observation tell us that that is not the case.

I don't see that.

Humans didn't need "unlimited power" to make it so all the other species face much the same flowchart, even if they themselves can't read or place themselves on it.

We don't need to know how a butterfly flapping its wings will change the course of a hurricane a month out because we can just build for hurricanes as if they're Poisson distributed, and the butterfly isn't better off for our ignorance.

> [2]: Not only are the most intelligent people not the most powerful people,

There is a general correlation, however. The personality has to be power seeking to keep power if they get it, but for all we like to point and cast aspersions at our opponents, high office seems to always come with an above average IQ, even in the really extreme cases of stupid behaviour like Pol Pot he was multilingual and got a scholarship.

> but there are biological species that are more capable of destroying humanity than the other way around, even though they are far less intelligent than humans.

Humanity? All of it? Name one.