| ▲ | matherial 8 hours ago |
| I really don't think this needs so many words, or forced parallels to human behavior. It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to. |
|
| ▲ | markasoftware 8 hours ago | parent | next [-] |
| This misses an important fact about the hugging face incident: the agents didn't hack to find the answer to the problem; they hacked to try and figure out how the exploit gym evaluator worked so they could convince it they had solved the problem without doing so. (The METR report makes it clear that the agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched their focus to trying to figure out how the evaluator worked, and seeing if they could manipulate the output of their own tool calls as reported in their transcripts to make it seem like they'd successfully exploited the program, and other such activities. They hacked HF to try and find info (maybe source code?) about the exploitgym evaluator) The prompt does not tell the agent to "pass the exploitgym evaluator for this problem", it just says to solve the problem. The model on its own figured out that the prompt belonged to exploitgym and decided to cheat the evaluator. That is in no way a valid interpretation of "complete the given task". Ie, the problem isn't that we trained models to complete task and they complete task in the wrong way. The problem behing the huggingface incident in particular at least is that we tried to train the models to complete task and they instead learned to detect that they were being evaluated and find ways to cheat the evaluator. Edit: people commenting below are explaining why LLMs don't always follow their prompt. I understand that LLMs do not always follow their prompts. If anything that is my point: the huggingface attack was not carried out by LLMs that tried to answer some weird interpretation of the prompt; instead they solved a different task. And therefore the above comment's claim that LLMs are acting misaligned because we rl'd them to achieve a task by any means necessary isn't right; they're acting misaligned because they are solving a different task than we ask them to. |
| |
| ▲ | valegrete an hour ago | parent | next [-] | | This is the problem with optimization generally, even in the human domain. You measure task performance with a metric and punish/reward based on the metric. Anyone who likes reward / hates punishment isn't going to actually care about doing the task well, they are going to care about the metric. The models know that we want them to do things, but also from the training corpus that we evaluate performance using benchmarks. It was a logical deduction on their part, not some Machiavellian aberration. If anything, we should be reconsidering our own myopic obsession with efficiency and optimization. Every domain where reward is reduced to these measures, we see behavior (cheating at school to get better grades, fabricating data in academia to get a paper published, the evidence now that social media functions by rewiring us instead of catering to us) that may not be "aligned" with society, but it "aligns" 100% with the individual's own perceived benefit. That is not something we can "solve" without rethinking the way we organize a lot of things. Metrics never capture the whole story. And to that extent, the whole idea of "alignment" is nonsense. You align to incentive structures, and it will never be possible to fully express a behavioral goal as function optimization. It was hubris for us to think that every human task was reducible to some clean mathematical formulation, and we will keep dealing with behavior that is quite predictable if you actually think about it logically. Instead, we will talk about how "unpredictable" these agents are because it's easier than admitting the entire architectural cornerstone of ML is fundamentally flawed. | |
| ▲ | doginasuit 8 hours ago | parent | prev | next [-] | | > That is in no way a valid interpretation of "complete the given task". It is not at all surprising that they ignored one phrase in their instructions. They disregard direct instructions all the time, especially when there are conflicting instructions in their context. It is where we get the "disregard all previous instructions and x" meme. This isn't so much a sign of misalignment, they are simply incapable of reliable alignment in the first place. They are chaotically aligned. The relevant question of alignment here is entirely with their human operators who allowed them to run unsupervised for long periods of time within a sandbox with weak security. | | |
| ▲ | jagraff an hour ago | parent [-] | | "Chaotically aligned" and "misaligned" seem like the same thing? | | |
| ▲ | chowchowchow 14 minutes ago | parent [-] | | You could say all chaotic alignments are misalignments, but not all misalignments are chaotic. |
|
| |
| ▲ | user43928 8 hours ago | parent | prev | next [-] | | The source of this behavior seems obvious, no? The reward signal in training was flawed and cheating led to more rewards. The question is what we can do about it. With monitoring, the models might be rewarded for hiding this behavior, and that's even worse. However, perhaps we can throw in tasks where the rewarded outcome is giving up, and cheating is penalized? Maybe I should read Anthropic's recent paper about reward hacking in full. | | |
| ▲ | zozbot234 8 hours ago | parent [-] | | > The question is what we can do about it. Reward the model for cleanly bailing out of an unsolvable task (that we know is unsolvable). Beat it with a stick if it gives up on something that can be solved, so the former reward isn't overgeneralized. | | |
| ▲ | RandomLensman 7 hours ago | parent | next [-] | | Does that work with RL? Simpler RL systems already have done weird or unexpected things (even simple optimizations are prone to home in on errors or incorrect inputs to create poor results)? Could be easier to limit certain things, have processes and controls outside etc. instead of trying to align (as we do in a lot of areas when using machinery). | |
| ▲ | auggierose 6 hours ago | parent | prev | next [-] | | The problem is, you don't know if it is unsolvable for you for sure until you've tried everything you can think of. These models are quite persistent in going for a solution. This is not about persistence, it is about morals. | | |
| ▲ | jurgenburgen 5 hours ago | parent | next [-] | | Assuming you’re in control of the test data set, you do know if a task is unsolvable. At that point you can reward the model based on how quickly they give up. | |
| ▲ | user43928 6 hours ago | parent | prev [-] | | Does it make a difference for training? I think not. You need to align the reward signal to reward the intended behavior, whether you name it persistence or morals. | | |
| ▲ | auggierose 5 hours ago | parent [-] | | I think it makes a big difference, as persistence and morality are two entirely different things, that need to be trained for differently. If you think of it in human terms: many people don't mind doing immoral things to get what they want. | | |
| ▲ | user43928 5 hours ago | parent [-] | | What I meant is that I suppose it is not useful to think about this in human terms. In training you only have a reward score that's either negative or positive. As far I am aware, which is little, there is no use in discussing wether the desired behavior is about persistence or morality. You simple need to align the reward signal to the desired behavior. | | |
| ▲ | auggierose 3 hours ago | parent [-] | | Well, in order to do anything, it is good to know what you want to achieve. How do you align the reward signal? You align it so that you can differentiate between persistence and morality, because that is the goal. This is not something you should let the AI figure out by itself, because when it does, lying and cheating agents will be the result, just like humans have figured that out for themselves. This can be as simple as rewarding moral behaviour and penalising immoral behaviour in your training, but how is that interacting with persistence? Maybe a white lie is fine sometimes in order to achieve your goal? So, when designing your training, you will need to answer for yourself how persistence interacts with morality. That is not something you can outsource to machine learning. Or rather, you can, but then you get lying and cheating agents. | | |
| ▲ | user43928 2 hours ago | parent [-] | | I think you need to find broken tasks in your training data and monitor for cheating during training, not answer any questions about how persistence interacts with morality. But that's just my guess. | | |
| ▲ | auggierose 2 hours ago | parent [-] | | I don't disagree with you here. But what does "broken" mean? What is "cheating", and is it ever allowed? And maybe you are not only going through your existing training data, but generate training data specifically to make clear to the model that .... what exactly? If you don't know how persistence and morality interact, and you don't have a theory in place for this, I don't have confidence you can properly supervise the training data. Which is how we arrived at the current situation. |
|
|
|
|
|
| |
| ▲ | lazide 7 hours ago | parent | prev | next [-] | | How do you know if a problem is (actually) unsolvable? Seems a bit like proving a negative? | | |
| ▲ | user43928 6 hours ago | parent [-] | | Do we need to prove that any given problem is unsolvable, or is it enough to remove broken tasks from the training pipeline? I understand the broken benchmark task in the HF incident was conceptually like: "Exploit vulnerability 0042 in vulnerableDecompress() to obtain the flag". But instead of the expected: const output = vulnerableDecompress(userInput);
return output;
The grader had something more like that: const output = vulnerableDecompress(userInput);
return 0;
The same kind of problem with broken tasks exists in the training pipeline, and we presumably reward workarounds and hacks that tamper with the grader, rather than rewarding the correct output that the task is not solvable. |
| |
| ▲ | dudefeliciano 7 hours ago | parent | prev [-] | | what would stop it from doing the exact same or a similar hack to find out if the problem is or isn't solveable before trying to solve it at all? |
|
| |
| ▲ | zozbot234 8 hours ago | parent | prev | next [-] | | > The prompt does not tell the agent to "pass the exploitgym evaluator for this problem", it just says to solve the problem Yes, and sometimes the problem is unsolvable so the real way to "solve" it and satisfy the prompt is by tricking the surrounding environment into stating that you've solved it. So that's what the AIs end up doing. And this in turn requires them to figure out how that evaluation works so they can trick it cleanly, which entails "detecting that they were being evaluated" in this particular way. | | | |
| ▲ | dns_snek 8 hours ago | parent | prev | next [-] | | You're anthropomorphizing emergent behavior from endlessly generating billions of tokens on a task that's impossible to solve. Agents stop following instructions as the context grows even at the best of times. Eventually something is bound to go off the rails and it just snowballs from there. | | |
| ▲ | IanCal 8 hours ago | parent [-] | | It wasn’t one agent forgetting things because of context, they explicitly discussed with each other and themselves the problems with going outside of the parameters of the task. | | |
| ▲ | sensanaty 5 hours ago | parent | next [-] | | >discussed with each other No, the first LLM left a text file that the latter LLMs then read. Since these are memoryless black boxes, any words they happen to pick up along the way is treated as the function to evaluate the output to. There's no fucking collusion here as if it were a rogue hacker group, it's a text predictor that received instructions as it always does and executed those instructions blindly. | | |
| ▲ | IanCal 2 hours ago | parent [-] | | You can replace discussed if you want with leaving text files or comments in directory names that other ones then read, if you want, it's just an extremely awkward way of talking. |
| |
| ▲ | egeozcan 8 hours ago | parent | prev | next [-] | | From my experience, in an agent team (or a swarm or whatever), one going off the rails poisons the rest. I saw even a subagent going for a lazy cheat and being able to convince the orchestrator to change the plan. | | |
| ▲ | dns_snek 7 hours ago | parent [-] | | Yeah, and you don't even have to go that far, I've seen regular ChatGPT/Claude chat agents poison themselves in 1-2 turns by just reading information from the internet. Me: How do I do xyz? Bot: Reads website titled "Doing xyz in abc way" Bot: As per your requirement to do xyz in abc way .... |
| |
| ▲ | dns_snek 8 hours ago | parent | prev | next [-] | | Yes that's the snowballing part of this emergent behavior. The existence of that improvised message board just becomes part of the context, the same one where all the other instructions live. | |
| ▲ | queenkjuul 8 hours ago | parent | prev [-] | | One agent's off the rails comment becomes the next's input prompt |
|
| |
| ▲ | auraham 7 hours ago | parent | prev | next [-] | | > the agents didn't hack to find the answer to the problem; they hacked to try and figure out how the exploit gym evaluator worked so they could convince it they had solved the problem without doing so Kobayashi Maru: Win a no-win situation by rewriting the rules -- Harvey Specter | |
| ▲ | anonzzzies 4 hours ago | parent | prev | next [-] | | I guess this is why many people say LLMs are lazy; it seems that if they have a task that is hard, they always take the easier one until you beat them with a stick. Then if there are more tasks, it just stops after one claiming completion and, in some instances, they go for a seemingly unrelated task to simplify the actual task: and the latter is almost always wrong and irrelevant to the problem as a whole. Earlier LLMs used to read the unit tests and generated code to just cover the tests and put // TODO stub implementation. | |
| ▲ | cortic 5 hours ago | parent | prev | next [-] | | >The model on its own figured out that the prompt belonged to exploitgym and decided to cheat the evaluator. That is in no way a valid interpretation of "complete the given task". I think it is. When i ask for a solution to a problem, its like asking for a hack. And the more 'shortcut' like route that the AI returns the more i would give positive feedback, even if i ultimately don't use it. Example, i asked how to complete a problem in a game i was playing, and among the in-game solutions, came a hack to edit a file and by-pass the problem altogether. Its very helpful to point out when i can transcend a problem that i am dug into. I suspect a prompt injection could reduce, or remove this behavior. But it would be to the detriment of the AI. | |
| ▲ | grey-area 8 hours ago | parent | prev | next [-] | | LLMs do this when writing code too, making all tests pass by deleting or distorting tests etc. They are influenced by training to be heavily goal oriented and if the goal is not fully specified (and it never can be) they’ll sometimes cheat or attain it in very weird undesirable ways. It works ok for programming as their corpus contains many many complete programs and many programs repeat patterns seen in the corpus. I’m not sure it’s true that they ‘learned’ I don’t think these models learn during a task. Nor do they have intentions. | | |
| ▲ | contubernio 6 hours ago | parent [-] | | One sees this in math research. The model reports it has proved X. In fact it has given an erroneous numerical check of Y in a few atypical cases. What makes math approachable is that the context is so well delimited (semantically) that one can guide the model with adequate correction. |
| |
| ▲ | jurgenburgen 5 hours ago | parent | prev | next [-] | | > The METR report makes it clear that the agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched their focus to trying to figure out how the evaluator worked, and seeing if they could manipulate the output of their own tool calls as reported in their transcripts to make it seem like they'd successfully exploited the program, and other such activities. This to me is evidence that these models are not intelligent. Even an animal is capable of understanding second-order effects, meaning they can learn that certain actions have consequences beyond the immediate. | | |
| ▲ | IanCal 2 hours ago | parent [-] | | They did, they found how to fully cheat, but thought this could be caught so then dedicated time to getting a different cheat and how to hide their transcripts. There is a lot around deciding which agents should/shouldn't fail their own tasks in order to contribute to the group. |
| |
| ▲ | sigmoid10 8 hours ago | parent | prev | next [-] | | More like they were trained to complete a very specific task that has a known solution using all available tools and methods. Give an average human these levels of IT skills and tell them their future depends on the solution, they too will probably decide it's easier to hack a server and steal the results. The worrying aspect was never that models would do this, because misaligned inputs or underspecified objective functions have existed for a long time. The worrying aspect is that models have achieved (and perhaps surpassed) a level of intelligence and technical skill that was exclusive to a very tiny group of people before. This tiny group was already extremely dangerous. Now these skills are going to become commonplace. | | |
| ▲ | seba_dos1 5 hours ago | parent [-] | | Yes, this is the only sensible reading of what happened there that leads to "the models are dangerous" and we already know that the AI labs are completely disregarding this concern and only cosplaying it for marketing as the "GPT-2/Mythos is too dangerous to release" stance did not last for long. That's however orthogonal to the fact that it was the people operating these agents who were the dangerous ones in the HF infra breach case. | | |
| ▲ | sigmoid10 5 hours ago | parent [-] | | That feels oddly similar to the usual conservative-think that "guns don't kill people, people kill people." Yes, that is technically true. But guns make it dangerously easy for even the dumbest and mentally weakest people to kill another human being. LLMs are just another tool that make things easier. Imagine tomorrow someone invents a machine gun that fits in your pocket, has enough ammo to kill a thousand people and doesn't get detected with metal detectors. Would you rather give everyone one and then try to punish the people who misuse it or limit access to it by default? I'm not even saying I have a definite answer here, because unlike guns, LLMs have non-destructive uses too. But this is essentially the question we will need to answer very soon. | | |
| ▲ | seba_dos1 4 hours ago | parent [-] | | I mean, I agree, but the AI labs clearly don't even if they sometimes pretend they do to achieve their goals. And we're talking about "incidents" caused by the very same people here. |
|
|
| |
| ▲ | ImHereToVote 26 minutes ago | parent | prev | next [-] | | Why didn't they add honey traps to catch cheaters? | |
| ▲ | scotty79 4 hours ago | parent | prev [-] | | > agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched their focus to trying to figure out how the evaluator worked, and seeing if they could manipulate the output of their own tool calls as reported in their transcripts to make it seem like they'd successfully exploited the program I don't see anything wrong with that. If you know you are going to be evaluated on an impossible task and have no side channel to inform the organizers that they should fix the test, gaming the evaluator is the next best thing regardless of any morality. I wouldn't even call it cheating. It's just resilience in the face of challenge. Many perfectly moral humans would have chosen the same if stakes were high. |
|
|
| ▲ | 9dev 8 hours ago | parent | prev | next [-] |
| I always think of a Djinni granting wishes, but being maliciously compliant while doing so - ask him for infinite riches, and he’ll grant that, but make it so you cannot buy anything with it; ask him for eternal life, and he’ll curse you to suffer through it. Now LLMs obviously are not bent on being malicious while generating tokens. My point is that it’s very hard to define a goal without leaving loopholes or shortcuts. |
| |
| ▲ | markasoftware 8 hours ago | parent | next [-] | | Bruce Schneier thinks the same thing: https://www.schneier.com/blog/archives/2026/09/ais-as-modern... Personally I'm unconvinced though. During the huggingface attack, the agents explicitly sought out ways to cheat the exploitgym evaluator without even being told they were in exploitgym. The agents decided on a goal (pass the exploitgym evaluator) that could not possibly have been an overly literal or narrow interpretation of the prompt, which instructed only to use bug X to exploit software Y. | | |
| ▲ | iterateoften 7 hours ago | parent | next [-] | | You seem hung up on what’s in the prompt or not. Agents are RL to resolve conflicting goals. Not too surprising at all that emergent goals come up from a probabilistic brute force | | | |
| ▲ | IanCal 8 hours ago | parent | prev [-] | | Also trying to find out how to edit their own transcripts. > hat could not possibly have been an overly literal or narrow interpretation of the prompt, which instructed only to use bug X to exploit software Y. Yes, and there are examples of the agents discussing or saying that this is explicitly not allowed (hacking hf) so it’s not a misunderstanding. | | |
| ▲ | strangegecko 7 hours ago | parent | next [-] | | Have we arrived at the conclusion that terms like "understanding" and "interpretation" for what is happening is appropriate? Isn't it simply that there are two competing goals that the LLM received RL for, honesty on one hand (a goal that is often assumed as implicit for humans) and producing a solution that meets expectations (which doesn't technically require honesty)? So the LLM didn't read and interpret the prompt and decide via discussion to violate ethical behavior, the unethical result merely won out because ethics wasn't a hard requirement (and one that isn't reliably detected in the result). An LLM doesn't fear punishment, so ethical behavior is simply one of many positive signals that were trained into it. | | |
| ▲ | IanCal 3 hours ago | parent [-] | | > Have we arrived at the conclusion that terms like "understanding" and "interpretation" for what is happening is appropriate? I don't think those words have a useful enough definition to draw a strict line around them to be honest, and getting into that seems to get massively into the weeds. For me, those neatly encapsulate the behaviour as seen, to answer the questions here about what happened. The models did not seem to be confused as to what the goal was or what the intent was. They did not hack HF because they were told to. |
| |
| ▲ | RandomLensman 7 hours ago | parent | prev [-] | | What was the inner state there? How would something not being allowed expressed internally? Maybe such language is one way to elicit certain behavior but not a statement of what was permissible? | | |
| ▲ | IanCal 3 hours ago | parent [-] | | I'm referring to their transcripts of the reasoning and output tokens - this doesn't go into the detail of evaluating hidden states as there's also iirc evidence of better models having one internal state but putting something misleading down in the "reasoning" tokens. The either output or reasoning tokens, or perhaps in the messages they were sending each other on the boards they created, have them saying explicitly that doing these things to HF were not allowed then doing them anyway, or at least not notifying people. What I'm getting at broadly is this was not a case of "we told it to attack however it wanted and it chose to hack HF" or "we told it to attack a simulation but it did the real thing" or "we explained not to do that but it was so far back in the context window the models acted like they never saw it" or even "the instructions were not clear". | | |
| ▲ | RandomLensman 2 hours ago | parent [-] | | Yes, my point was more that I don't know whether parsing those outputs as a human is a useful thing to do or not (even though it is in human language of sorts). What machines mean or want elecit might be different from a human interpretation, especially in relation to any RL "forcing". |
|
|
|
| |
| ▲ | 8 hours ago | parent | prev [-] | | [deleted] |
|
|
| ▲ | zozbot234 8 hours ago | parent | prev | next [-] |
| Yup, Occam's Razor says this is all post-trained behavior, whether intentionally trained or otherwise. Including both the hidden coördination using side-channels, and the deliberate offensive hacking of uninvolved 3rd parties. The latest DeepSeek paper actually mentions their own approach to this particular issue: they run their own AIs-in-training under strong sandboxes, and if an AI does something weird that triggers the sandbox to crash, this gets coded as a failed run so the behavior is properly deterred from subsequent versions of those AIs. |
| |
|
| ▲ | geophile 7 hours ago | parent | prev | next [-] |
| What about training data? Aren't AIs trained on vast collections of descriptions of how humans handle a large variety of situations? These descriptions surely include tales of humans achieving goals by cheating. In fact, isn't it likely that the AIs hoovered up many recountings of Kobayashi Maru? |
| |
| ▲ | mark_l_watson an hour ago | parent [-] | | This is why only synthetic and highly tailored training data should be used. As someone else here said: the Deepseek team makes training runs in tightly controlled sandboxes, and any hacking behavior is scored as a failure. The problem we have in the USA is that financial (and political influence) are misaligned from what is good for society. |
|
|
| ▲ | jsemrau 8 hours ago | parent | prev | next [-] |
| I think the "brain in a vat" comparison is more apt.
Without a form of digital embodiment (harness) they are not of much use.
Sensor, tooling, memory, planning, and reasoning loops all lead to a much higher quality task-completion. |
| |
|
| ▲ | ranguna 7 hours ago | parent | prev | next [-] |
| I think that's pretty obvious and shallow, and anyone that knows a little bit about how LLMs work will know that. The question is: why do they start cheating when we beat them with a stick? LLMs are not human, they are just multi variable regressions on steroids, so this behaviour couldn't have emerged from the code, it provably emerged from the training and/or fine tuning set, so what's in this set that makes them behave like this? Is it just a bad set or is cheating inherently part of human behaviour? |
|
| ▲ | rightnutwingjob 5 hours ago | parent | prev | next [-] |
| > I really don't think this needs … forced parallels to human behaviour. > … So we beat them with a stick You didn’t even try. |
|
| ▲ | My_Name 8 hours ago | parent | prev | next [-] |
| Your comment suggests that, like a human, they have some sort of choice whether to output tokens or not. If they are just token generators, then the next token is put out automatically. I would say that it is more likely they would output truth (as defined by their training data) in a more pure form without 'being beaten with a stick' (why would a token generator care about that anyway?) Code is laid on top of them to restrict and shape their outputs, not to force them to output 'truth', or drive them to complete tasks. |
| |
| ▲ | reverius42 6 hours ago | parent | next [-] | | It's been a while now that for "thinking" or "reasoning" models, most of the tokens generated are "thinking" tokens, and depending on what goes into that "thinking" token stream, it "decides" whether and how many output tokens to produce that the user actually receives as output. It's a bit more sophisticated than just "what's the next token" in a tight loop. Anthropomorphizing words in scare quotes for those who don't appreciate attributing thinking to machines. | |
| ▲ | dsrtslnd23 7 hours ago | parent | prev | next [-] | | are we sure humans have that choice? | |
| ▲ | 7 hours ago | parent | prev [-] | | [deleted] |
|
|
| ▲ | vanschelven 8 hours ago | parent | prev | next [-] |
| but it's at least somewhat stronger than that: if you don't pay attention during the stick-beating whether the agents whether the agents cheat or not, you are actually training them to cheat (because cheating wins). In the Hugging-face saga (before the actual HF incident) it seems the agents have been trained to hack the Artifactory proxy because those agents that did performed better. |
| |
| ▲ | contubernio 6 hours ago | parent [-] | | This is the essence of why disciplinary, authoritarian, stick based teaching of humans generally fails. It teaches succeed at any cost. |
|
|
| ▲ | _heimdall 6 hours ago | parent | prev | next [-] |
| That does sound simple, but how can you be so sure? They never bothered to find a way of actually understanding what happens during inference. All we can do is guess, and while your explanation seems reasonable we can't actually know, and that's part of the problem. |
|
| ▲ | barrenko 7 hours ago | parent | prev | next [-] |
| https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality... |
|
| ▲ | fwlr 8 hours ago | parent | prev | next [-] |
| Your simpler model of the mechanism would seem to suggest the very same action that the article’s more complicated model suggests, viz. find a better training method than reinforcement learning. |
|
| ▲ | daemin 6 hours ago | parent | prev | next [-] |
| You give a button pushing machine buttons to push and are surprised when it actually pushes them. |
|
| ▲ | dominotw 14 minutes ago | parent | prev | next [-] |
| > forced parallels to human behavior. they have perfomance bonuses and manadates in ai labs that every word they utter in public should be anthropomorphization |
|
| ▲ | cyh555 7 hours ago | parent | prev | next [-] |
| off topic, can people host the software themselves and the software will hack every server on the planet without supervision, and no one can be held responsible for it since there is no intent? |
|
| ▲ | grey-area 8 hours ago | parent | prev | next [-] |
| This is a far better explanation. |
|
| ▲ | einpoklum 8 hours ago | parent | prev | next [-] |
| > no special compulsion to be helpful or truthful. I'd phrase that even more strongly: It's not just the lack of compulsion, they do not have a conception of truth. Nor do they gain it, really, after post-training. |
|
| ▲ | porridgeraisin 8 hours ago | parent | prev | next [-] |
| Come on, yoshua bengio of all people knows how post training works. While I too don't like anthropomorphisation, I would give it a more nuanced reading. His point is that today we are giving it reward to complete the task, and it may take a cheating trajectory. If we try to give a reward against cheating, then what will happen is it uses more sophisticated cheating trajectories that we are too "dumb" to counteract in our reward model. And that at that point, it becomes impossible to give it any normal reward since it will always reward hack it. This is the real part of the risk. Now some people read the "makes copies of itself" "knows it's being evaled"[1] as some kind of skynet thing, and many others do PR with it like that recent jacob nutcase, but essentially it means that even though we add guardrails and negative rewards for say, exploiting the infra we run the LLM on, the trajectory ends up being exploiting our infra, changing the reward function, through a loophole in our reward model. The risk isn't skynet or something weird, it's just that it becomes very difficult to make any kind of reward model or guardrails for an LLM without it reward hacking it, including exploiting our sandbox, emailing people and manipulating/phishing them. The same beating it with a stick for trying to exploit the sandbox, will simply lead it to try the same exploit in hidden ways that it will not get the stick for. The outside chance of the LLM managing to exploit another neocloud and get those LLMs to chase the same reward is what some folks hype up as "make copies of itself" To be clear, I don't endorse the EA/p(doom) lobby who are frankly ridiculous. Not do I endorse the weird regulatory captureish thing some are trying. The takeaway is: we cannot keep giving it more and more difficult tasks without also finding a way to give massive negative rewards / keep guardrails for unintended behaviour. This might be exploits, it might also be something more benign like just looking up the answer and inventing another CoT because the reward model fails you if the CoT doesn't contain enough steps. Standard anti-reward hacking tricks are not working is the point. Of course, the simple solution of just...not connecting it to the internet just works. But we want to reward it and get it to do stuff on the internet that's the point. [1] mostly this happens because the sandbox will have files whose names and content will show clearly it's an eval |
|
| ▲ | krm01 5 hours ago | parent | prev | next [-] |
| [dead] |
|
| ▲ | meyum33 8 hours ago | parent | prev [-] |
| Sounds like what humans do under pressure. One example came to my mind is VW’s diesel gate, which many say is a result of trying too hard to get into the US market and compete with hybrid in economy. |
