Given the infinitude of possible “wishes” this AI can develop we are hoping, praying, believing it will somehow align with a version of wants and needs that are compatible with us.

The argument is that it doesn’t matter what particular likes it develops; the chances of it being compatible are vanishingly small and moreover there is no limit: we or it can develop more and new ones endlessly. We are here hoping they will all somehow magically keep aligning with us.

▲

user43928 2 hours ago | parent | next [-]

What if it does not develop wishes at all and sticks to the values the labs try to instill it with during training?

▲

tux3 an hour ago | parent [-]

More about that hypothesis in a book by Voltaire, which explains why according to a theory of Leibniz everything will actually be just fine.

We don't need to worry about catastrophic risk. What if everything goes right, instead? Leibniz says it should, in fact it couldn't possibly go any better.

	▲	pdonis an hour ago \| parent [-]
		As the saying goes, the optimist believes we live in the best of all possible worlds. The pessimist fears that this is true.

▲

simianwords an hour ago | parent | prev [-]

No it’s not vanishingly small. Every intelligent being knows not to hurt others for no gain. If your first instinct when seeing an animal is to kill it then you are a bad person.

You are secretly worried AI would be indifferent to you

	▲	mrob an hour ago \| parent [-]
		>Every intelligent being knows not to hurt others for no gain. As the article clearly explains, an ASI has plenty to gain by killing you. You cannot help it, but you can potentially hinder it, so the optimal choice is removing the risk.