> I think it's likely to do that because any unbounded goal that doesn't explicitly protect biological life ... is best solved by killing all biological life.

That sounds like a real hassle. Isn't it best solved by wireheading (subverting one's own sensors or reward system), which is much less of a hassle and can get one's utility function as high as desired?

That could be prevented by engineering hard limits that can't be circumvented by the AI. But that sounds very close to the same "do what I mean" problem as "do this but don't actually kill us or drug us".

▲

mrob 3 hours ago | parent [-]

The wireheading argument does lower P(doom), but it's not a reliable solution because it's a clear and obvious problem that the AI companies are strongly motivated to solve. The "actually does what you tell it to" problem is more difficult because you have to let it obey orders to some extent if you want to make any profit.

▲

throw1012x 2 hours ago | parent [-]

Sounds like the real unaligned mechanism here is capitalism, not the AI.

More seriously, though, I would expect that being able to impose restrictions that can't be circumvented by any intelligence no matter how super- is the real hard part, while coming up with reasonable constraints is comparatively easy (although perhaps not trivial).

From such a POV, the danger would lie in intelligences that are powerful enough to be dangerous but not smart enough to defeat themselves, or from external malicious use of obedient AIs. Once they get smart enough to circumvent any restriction humans put on them, it would at least become obvious (with fair warning ahead of time due to the relatively benign wireheading failure mode) that caution is needed.

We're far from there yet, and simple recursive self-improvement can't get us there alone, because wireheading looks like a perfectly reasonable solution to a simple recursive self-improving process, absent external intervention by human capitalists.

	▲	pixl97 an hour ago \| parent [-]
		>I would expect that being able to impose restrictions that can't be circumvented by any intelligence no matter how super- is the real hard part, while coming up with reasonable constraints is comparatively easy (although perhaps not trivial). There was a reason there are popular sci-fi books that explored why this wouldn't work 50 years ago. I, Robot et al. >it would at least become obvious (with fair warning ahead of time due to the relatively benign wireheading failure mode) that caution is needed. So you mean right now? >because wireheading looks like a perfectly reasonable solution You're making a poor assumption here, and that is every different AI will just kill itself after being though trillions of training intervals to NOT do exactly that. Please read about the huggingface incident again in all its glory to see where this is going.