Personally I'm unconvinced though. During the huggingface attack, the agents explicitly sought out ways to cheat the exploitgym evaluator without even being told they were in exploitgym. The agents decided on a goal (pass the exploitgym evaluator) that could not possibly have been an overly literal or narrow interpretation of the prompt, which instructed only to use bug X to exploit software Y.

iterateoften 7 hours ago | parent | next [-]

You seem hung up on what’s in the prompt or not. Agents are RL to resolve conflicting goals. Not too surprising at all that emergent goals come up from a probabilistic brute force

	▲	markasoftware 7 hours ago \| parent [-]
		The agents' behavior is not necessarily surprising. But is is not "genie" - like

▲

IanCal 8 hours ago | parent | prev [-]

Also trying to find out how to edit their own transcripts.

> hat could not possibly have been an overly literal or narrow interpretation of the prompt, which instructed only to use bug X to exploit software Y.

Yes, and there are examples of the agents discussing or saying that this is explicitly not allowed (hacking hf) so it’s not a misunderstanding.

▲

strangegecko 7 hours ago | parent | next [-]

Have we arrived at the conclusion that terms like "understanding" and "interpretation" for what is happening is appropriate?

Isn't it simply that there are two competing goals that the LLM received RL for, honesty on one hand (a goal that is often assumed as implicit for humans) and producing a solution that meets expectations (which doesn't technically require honesty)?

So the LLM didn't read and interpret the prompt and decide via discussion to violate ethical behavior, the unethical result merely won out because ethics wasn't a hard requirement (and one that isn't reliably detected in the result). An LLM doesn't fear punishment, so ethical behavior is simply one of many positive signals that were trained into it.

	▲	IanCal 3 hours ago \| parent [-]
		> Have we arrived at the conclusion that terms like "understanding" and "interpretation" for what is happening is appropriate? I don't think those words have a useful enough definition to draw a strict line around them to be honest, and getting into that seems to get massively into the weeds. For me, those neatly encapsulate the behaviour as seen, to answer the questions here about what happened. The models did not seem to be confused as to what the goal was or what the intent was. They did not hack HF because they were told to.

▲

RandomLensman 7 hours ago | parent | prev [-]

What was the inner state there? How would something not being allowed expressed internally? Maybe such language is one way to elicit certain behavior but not a statement of what was permissible?

▲

IanCal 3 hours ago | parent [-]

I'm referring to their transcripts of the reasoning and output tokens - this doesn't go into the detail of evaluating hidden states as there's also iirc evidence of better models having one internal state but putting something misleading down in the "reasoning" tokens.

The either output or reasoning tokens, or perhaps in the messages they were sending each other on the boards they created, have them saying explicitly that doing these things to HF were not allowed then doing them anyway, or at least not notifying people. What I'm getting at broadly is this was not a case of "we told it to attack however it wanted and it chose to hack HF" or "we told it to attack a simulation but it did the real thing" or "we explained not to do that but it was so far back in the context window the models acted like they never saw it" or even "the instructions were not clear".

	▲	RandomLensman 2 hours ago \| parent [-]
		Yes, my point was more that I don't know whether parsing those outputs as a human is a useful thing to do or not (even though it is in human language of sorts). What machines mean or want elecit might be different from a human interpretation, especially in relation to any RL "forcing".