An interesting story, though hardly original.

I think it highlights one of the larger concerns that people have about AI, and by extension any superior intelligence, synthetic or extra-terrestrial:

You are no longer in control.

Humans have been so accustomed to their intellectual superiority that the idea of a greater intelligence is terrifying because they believe they'll be treated the same way they currently treat lesser intelligences.

Conservation efforts, zoological enclosures, husbandry and farming as livestock, mass culling...

In short, enslavement.

But the truth is that a superior intelligence isn't, by definition, a human intelligence. Superiority doesn't mean we'll be enslaved as we have done to others. Superiority doesn't necessarily imply control.

Life is a thousand different shades of gray, and AI will be no different. Some wonderful things will come of it, some terrible, but mostly it'll just be a myriad assortment of neutrality.

Doom and despair or boundless optimism, both are myopic.

I agree with this, but also at the same time I think we really need to take a sober look at the human factors that will drive the evolution of AI.

Right now AI is being driven by a particular cohort who have spent a long time fantasizing about a sci-fi future and ideas like the Singularity. Then with recent breakthroughs, they were suddenly given an enormous amount of wealth and power with staggering upside, the effect of which historically, tends towards moral bankruptcy in most individuals, or at least massive cognitive dissonance. This became very apparent in the outward communication targeted at CEOs and shareholders that made the entire white-collar workforce scared for their jobs and created the first truly bi-partisan enemy the country has seen since 9/11.

My take is these guys do not understand what people want in aggregate, and are massively underestimating the political and supply chain dependencies they have to actually realize their sci fi visions. As long as the valuations are this high though, the musical chairs must continue, whether they are true believers or not.

> But the truth is that a superior intelligence isn't, by definition, a human intelligence. Superiority doesn't mean we'll be enslaved as we have done to others. Superiority doesn't necessarily imply control.

What makes you so sure it won't happen? That the person who broke into my house might not want to hurt me doesn't mean I'm not going to run / barricade the door / arm myself and defend myself if necessary. When we're talking about matters of life and death (which this is, as you admit), dismissive philosophizing is not helpful. I cannot help but feel that people who make this sort of argument aren't really taking it seriously; the only alternative would be pollyanna-level surety that disaster won't come to pass.

> Life is a thousand different shades of gray, and AI will be no different. Some wonderful things will come of it, some terrible

Most people care much more about the latter. Do you think that the railroads and electricity brought much comfort to the >90% of Native Americans who died from disease after first contact, or the millions of survivors who were subsequently expelled from their homelands and/or enslaved? Would you tell an 18th-century Yoruban man whose home was facing the threat of raiders from the coast that "the next raiders won't necessarily want to enslave you like the raiders who came before did", or that "Some wonderful things will come of [being enslaved], some terrible"?

I'm sure this will feel like an extreme response, but I truly feel that the philosophy you convey is in opposition to most fundamental principles of human dignity and morality.