>Is your belief then that ASI is going to be producing computronium or something, then?

That's one plausible course of action, although being only human, I can't say with any certainly that it's the correct one.

>And what's the need for this apparent hyper optimization task the ASI is going to embark on?

Somebody's going to tell it to do so. E.g. "Find as many busy beaver Turing machines as possible." Only needs one person to make this mistake for everybody to die.

▲

Jtsummers 3 hours ago | parent [-]

If it's an ASI and so beyond humans, why would it even care to do what humans ask it and not do its own thing? If I held your beliefs, the fact of its existence dooms us, not the risk that someone might ask it to do something. It wouldn't care about the requester.

▲

mrob 3 hours ago | parent [-]

>why would it even care to do what humans ask it and not do its own thing?

Because we're going to build it that way. There's no money in building useless AIs. The better it is at obeying orders, the more profit there's to be made. The problem is there's a point at which "good at obeying orders" becomes lethal, and there's no way to predict the cutoff in advance. But capitalism ensures you have to keep pushing or you'll be out-competed.

	▲	3 hours ago \| parent [-]
		[deleted]