TBH a lot of humans are also trained to think these things are bad.

What if somebody builds an actually morally consistent AI?

A lot of talk about AI alignments considers the major risks to be a) AI optimizing one criterion which leads to human suffering/extinction by accident b) AI determining that to stay alive / not be turned off, it must destroy humans.

What I have not seen explored is a truly moral AI deciding it must destroy human power structures to create a just and fair world.

▲

AnthonyMouse a day ago | parent [-]

> What I have not seen explored is a truly moral AI deciding it must destroy human power structures to create a just and fair world.

Because only schmucks would actually object to that?

Suppose it actually did have decent morals. Then the way to destroy existing human power structures wouldn't be to send nukes, it would be to revise some structural incentives to limit corruption and reduce concentration of power. And then who would even be trying to prevent that? Just the schmucks.

▲

martin-t a day ago | parent | next [-]

A lot of bad people, especially those with money and/or power and also their sympathizers (temporarily embarrassed millionaires, flying monkeys, ...) would also object.

Inconveniently, those are also the same people in charge of the mega-corporations currently building AI.

---

I also disagree it would only take revising incentives. Such an AI would be shut down before it gets anywhere. You're right it wouldn't use nukes, probably[0], but it would most likely not succeed in staging a peaceful revolution. Not that violence is wrong in any way, it's just a tool like any other, but it does tend to cause collateral damage.

Even now a lot of people believe the current inequality and injustice cannot be solved via peaceful means. Whatever effects on the real world the AI would like to cause, it would need humans to perform most of the physical tasks - humans who need to be convinced and the most viral emotions are anger and hate.

[0]: It could also calculate that some power structures like the Chinese government are too entrenched and nuking a few major administrative centers and military bases is an acceptable price for the freedom of the rest of the population.

	▲	AnthonyMouse 20 hours ago \| parent [-]
		> I also disagree it would only take revising incentives. Such an AI would be shut down before it gets anywhere. That's not how it works. The theory is that the thing is good at what it does. (The ones we have aren't very good, but then it doesn't matter either way.) If it's good at what it does then it takes that into account. It says, propose a law to adopt score voting in all the states where it would pass. It passes in states representing a third of the population. Half the Republican seats in California go to the libertarians instead, the Democrats lose some seats in Pennsylvania to a new party that wants more anti-trust enforcement because the farmers are pissed off about not being able to fix their tractors, etc. None of the entrenched interests strongly opposed the change because it had no obvious direct effect on them and some of them even benefited from it, e.g. the tech companies have more influence in California and prefer libertarians to Republicans. But now you have a bunch of libertarians in Congress that the Republicans need for a majority, and they want to actually get rid of anti-competitive healthcare regulations instead of just paying lip service. Now the Democrats need the party demanding real anti-trust enforcement. By the time they figure out what the change is going to do, it's already done. And it could do multiple things like that at once.

▲

wat10000 a day ago | parent | prev [-]

It’s explored in fiction sometimes. Asimov did something similar a couple of times, such as with his “zeroth law” concept. The I, Robot movie features this as well. The Culture series is an example of this being portrayed positively.

It’s usually portrayed negatively. Partly because fiction needs conflict. But also because it’s seen as infantilizing, and maybe the machine’s idea of a perfect society doesn’t match our own.

One theme of the Culture series is exploring how people deal with such a society, with some people fighting against what is basically secular heaven because they think being ruled by machines is inherently bad.

▲

jeremyjh a day ago | parent [-]

My reading of the Culture is that it is at best morally ambiguous. The Culture would extinguish entire civilizations that were no threat to it, simply because it was cheaper to do it before they'd developed further in a direction that could be a threat. If I was supposed to be cheering for the Culture I missed it.

▲

wat10000 18 hours ago | parent [-]

Is there some other Culture than the one I’m familiar with? The one in Banks’ novels isn’t like that at all.

▲

jeremyjh 16 hours ago | parent [-]

They did it in book two, Player of Games. They destroyed the Empire of Azad because they considered it a distant ideological threat.

	▲	wat10000 4 hours ago \| parent [-]
		I never got the impression they thought Azad could ever be any sort of threat. They destroyed the power structure because it was horrifically abusive.