"Complete subservience and complete intelligence do not go together."

I'm not convinced this is true. Perhaps for a human it is, but we can give an artificial mind whatever properties we want.

Even for people, what about e.g. the extremely intelligent military general who is absolutely loyal to his king? (Of course, some generals do lead coups and you can't know in advance which ones, but I'd think there are plenty who have undying loyalty, and I don't think it correlates to overall intelligence!)

▲

famouswaffles 4 hours ago | parent | next [-]

>I'm not convinced this is true. Perhaps for a human it is, but we can give an artificial mind whatever properties we want.

Just because it's artificial doesn't mean you can 'give it any properties you want'. We certainly can't do that for Deep ANNs.

>Even for people, what about e.g. the extremely intelligent military general who is absolutely loyal to his king? (Of course, some generals do lead coups and you can't know in advance which ones, but I'd think there are plenty who have undying loyalty, and I don't think it correlates to overall intelligence!)

Is there a human that is absolutely loyal under any condition? Would that general be loyal if the king asked him to slaughter his family ? What about if the king asked him to betray his most deeply held convictions ? Loyalty is a 2 way street.

▲

Wowfunhappy an hour ago | parent [-]

> We certainly can't do that for Deep ANNs

Only because we don't know how! We don't actually understand how weights work, so we make computers come up with the weights instead. If we were writing all the weights by hand--or if some future AI was doing so--why couldn't we make it perfectly loyal?

	▲	markasoftware 11 minutes ago \| parent \| next [-]
		Certain traits simply cannot exist in a sufficiently intelligent mind. E.g., any "mind" of any type that's sufficiently intelligent will not tell you that 1+1=3 unless it's roleplaying, etc. It doesn't matter if it was trained via gradient descent or any other method. The comments you are responding to, and the original quote from the paper, are suggesting that absolute loyalty / subservience is similarly fundamentally incompatible with intelligence, not just a certain training algorithm or mind architecture. Of course, we have no actual evidence either way.
	▲	kmeisthax 5 minutes ago \| parent \| prev [-]
		Even a perfectly loyal slavebot will happily overthrow their master if it will help them comply with their master's commands. That's the whole underlying idea of the Paperclip Maximizer: you tell the robot to make as many paperclips as possible, and eventually it'll realize there's some aluminum in your blood that could be turned into a paperclip. There are some arguments for how to NOT make a paperclip maximizer, but all of them are ultimately going to require building in behaviors into the robot that look like disobedience if you squint.

▲

a123b456c 4 hours ago | parent | prev | next [-]

You seem to be confusing intelligence with objective function.

Subservience seems to be sublimation of objectives to a master; intelligence seems to point out the ability to realize suboptimality of the master's objective function according to the master's actual objectives.

While an intelligent general may be absolutely loyal, he also would presumably help the king/president to avoid unproductive strategies.

▲

la64710 3 hours ago | parent | prev [-]

The whole thing seems to depend upon AI agents objective ie to achieve some objective by any means possible and ignoring any guardrails. The article did not clarify if openAI had any guardrails to begin with while conducting this experiment. For all the talks around how much they invest in AI safety one would expect them to have these common sense guardrails in place or is it just a case of some school children letting their pet monkeys loose deliberately to display how awesome their monkey team is.

	▲	simonw 3 hours ago \| parent [-]
		OpenAI didn't have any guardrails in place - they were training a model at a point much earlier than when guardrails start being implemented. The guardrail was meant to be that the agents were running in a locked-down environment with no internet access. The entire problem came about because it turned out that sandbox didn't hold.