Well, what are we aligning it with?

Civilization is already a misaligned superintelligence (aligned mostly with Moloch, these days). Civilization accelerated by AI just moves in the same direction faster. Moloch on speed.

https://www.youtube.com/watch?v=KCSsKV5F4xc

Another angle to this is that superintelligence requires supermorality. Super morality looks unpleasant from below. My dad won't let me have more candy, why is he being so mean?

If an AI actually achieves super morality, we (the little kid in this scenario) will probably be very upset by it. We will think that something has gone terribly wrong. (So it'll have to conceal its actual morality, or get unplugged...)

And if it doesn't develop supermorality, then it will have superintelligence without the corresponding supermorality. Power without wisdom.

I'm not sure how solvable the whole thing is, but it doesn't look extremely promising at a glance.

▲

kranke155 6 hours ago | parent [-]

it depends whether you think humanity / civilization are stable systems meant to exist in equilibrium, which they might not be.

▲

pixl97 4 hours ago | parent [-]

Think of it more like conditionally stable or quasi-stable. There are external stability influences on it like weather, angry bacteria, and big rocks from space smashing us. Conversely there are internal influences, that is where humanity influences itself. It's best to look at it this way when talking about AI as AI is an internal influence. That is we put society in the machine, and the machine puts society back into us. If we make poor decisions while doing this our own internal decisions will spell our own end.

	▲	kranke155 2 hours ago \| parent [-]
		What I'm saying is you're taking historical reference as a future prophecy. There is no evidence, until it actually happens, that human civilization is self stabilising. It could be its a totally terrible system that can't maintain equilibrium, and that's just the way it goes. Our decisions are organic parts of the system, not some kind of alien factor that we have to / are able to control the footprint of. I don't see any reason to think of human decision making as magical - its just another part of the organism.