| ▲ | cmiles8 5 hours ago |
| Both can be true. It’s clear models are doing rogue things. But it’s also clear that the labs look like amateur hour with their technical security and testing practices. |
|
| ▲ | mrweasel 5 hours ago | parent | next [-] |
| It's is wild to see investors go crazy and hype the value of the AI labs when ever they talk about rogue agents. They completely ignore the possibility that these people don't know what they're doing operationally. |
| |
| ▲ | schrodinger 5 hours ago | parent | next [-] | | *rogue, rouge = red in French! | | | |
| ▲ | miohtama 5 hours ago | parent | prev | next [-] | | Same happened with FTX and co. during 2020 crypto boom. A lot of rogue stuff sloshing around, but investors were hyperventilating. | |
| ▲ | aesthesia 5 hours ago | parent | prev | next [-] | | People keep saying this, but I haven't seen any actual instances of investors saying that rogue agents mean AI labs are valuable. Do you have any examples? | | |
| ▲ | falserum 3 hours ago | parent [-] | | There is no such thing as bad publicity.
(Technically not true, but definitely applicable in this concrete case, as planned P/E ratios are crazy) |
| |
| ▲ | djierardi 4 hours ago | parent | prev | next [-] | | See https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-che... | |
| ▲ | viklove 5 hours ago | parent | prev [-] | | Investors are motivated by returns, not responsible decision-making or societal benefit. Our culture is rooted in rewarding greed and granting the greediest individuals power over the rest of us. Isn't this the inevitable end-state of unfettered capitalism? | | |
| ▲ | MentalM 3 hours ago | parent [-] | | > Isn't this the inevitable end-state of unfettered capitalism? Aren't capitalism built on exactly the opposite principles? It is not greed that rewarded, but the creation of societal goods, and power granted not to the greediest, but to the most useful individuals. |
|
|
|
| ▲ | lukewarm707 4 hours ago | parent | prev | next [-] |
| Only one is true. Per US law, Amazon v. Perplexity 2026, it is not possible for an AI model to go rogue. An AI model is a tool. The user directing the agent is responsible for the damages it causes. A so-called 'agent' does not have the free will, intent or ability, to decide what it does. Its freedom to make decisions can't be taken away because it does not have any freedom to make decisions. It can't be deterred, convicted or interned. It does not choose to conform; it does not choose to go rogue. |
| |
| ▲ | astine 3 hours ago | parent | next [-] | | It may be the case that companies are liable for anything their products do, but that doesn't mean that they can't go rogue. It just means that companies are liable when the AI models do so. Not that I think we should be credulous about stories where they go rogue. | | |
| ▲ | lukewarm707 3 hours ago | parent [-] | | If I make a model that always does the opposite of my instructions, it does not choose to go rogue by deviating from my instructions. It was set up to deviate! So the model "went rogue" but did exactly what I intended. The idea of an agent making a decision of its own "will" is an illusion. Variation in the output is caused by a random number generator. Random is not equal to free will. If I run the same agent with greedy sampling 100,000 times with the same inputs, it will think the same thoughts, and it will say the same things, 100,000 times. For example, if you use synthID or whatever 'watermark', the agent does not get a choice about what words it will vary to identify itself. An agent is whatever the person making it wants it to be; if I train an agent in Greek it will only know Greek. It will never speak English! | | |
| ▲ | dwaltrip 3 hours ago | parent | next [-] | | You need to account for reward hacking and unexpected outcomes from training. | | |
| ▲ | lukewarm707 3 hours ago | parent [-] | | If I make a cake, and it unexpectedly explodes in the oven, it does not mean that the cake itself developed the free will to explode. The model does not have free will because I did not predict what it would do! | | |
| ▲ | Sniffnoy 2 hours ago | parent | next [-] | | Why are you arguing over irrelevancies? When people say the models "went rogue", the part of that is relevant is that it did dangerous things that were not reasonably predictable in advance and cannot be easily prevented (except by not running it). Philosophical questions are irrelevant. if you agree that such models are dangerous because they may in response to instructions -- including in response to careless instructions, because people will be careless, that is unavoidable! -- do unpredictable bad things outside of expected bounds, then you agree on the essential points! Or really perhaps what I should say here is, the question that's really being argued over here isn't even the one I just said, rather it's one of appropriate action -- should such security breaches be solved by improving security, or by doing something about the models (including potentially just not running them)? (The answer may be both.) If you agree that the solution includes the latter, then you agree that the models are "going rogue" in the sense that is being argued over, regardless of philosophical issues! Don't turn a question of actual import into one of philosophical irrelevancies! If you object to the terminology, fine, but in that case, instead of turning the argument to philosophy, you should say "instead of saying 'going rogue', I would prefer to instead say <your preferred terminology for what's being described>, as that doesn't carry such philosophical baggage." | | |
| ▲ | lukewarm707 31 minutes ago | parent [-] | | > "Should such security breaches be solved by improving security, or by doing something about the models?" The philosophical question about free will is actually the grounding of the legal one. Sam wanted to be the best, so Sam accelerated to 120mph in his car and lost control. You are the judge: a) Sam is responsible. Convict Sam and suspend Sam's driving license. b) The car is responsible. Tell Sam to make the car safer. Capacity: was the car a legal person? Clearly no. But maybe at best the car in some way commited the physical act of going fast. However, that alone would not establish guilt. "The act is not culpable unless the mind is guilty" (mens rea). Did the car "consiously" disregard a duty of care by going 120mph? Did the car know that it was wrong to go 120mph and yet choose to do it anyway? On the question of free will, the car could not do otherwise. It it did not consciously, knowingly, purposely, or recklessly accelerate to that speed. The car did not have a guilty mind because it did not have a mind at all. So in terms of justice and resolution, none of the focus here is on the car. | | |
| ▲ | Sniffnoy 4 minutes ago | parent [-] | | Ah, I see. It appears we have an implicit disagreement of scale. I actually do disagree with much the more explicit content of your comment, but I am not going to bother to expand on that, because it is the less directly relevant of the two disagreements here, and it would take time to write that. The more relevant disagreement is that you are (I gather) assuming a context where justice and restitution are a sufficient response to the problem (or else you wouldn't be focusing on them to this extent, I am inferring). Like, where the incidents don't get that much bigger than what has already happened. So the question becomes how do we distribute the loss, and how do we disincentivize them. However, when the scale of a potential incident gets big enough, justice and deterrence are not sufficient; prevention becomes the far more relevant consideration. You may have noticed that many people are worried about potential human extinction as a result of AIs. In that case, prevention becomes the only consideration -- justice is useless. So there seems to be an implicit disagreement here between thinking of such events as being on a scale not too much larger than that which has already happened, being a new normal that the legal system will need to handle, vs thinking of such events as warning shots for something that requires prevention and not justice. So, while I also disagree with the more explicit content of your comment, the fact is I simply wasn't thinking about such matters in the first place -- I was talking about this in the frame of a problem that requires prevention, I wasn't thinking about the justice question at all. It honestly failed to occur to me that you might be thinking of this as a problem where justice suffices! |
|
| |
| ▲ | dwaltrip an hour ago | parent | prev [-] | | What do cakes have to do with LLMs and how they are trained? Not much? |
|
| |
| ▲ | astine an hour ago | parent | prev [-] | | I don't see any talking about agents "choosing" to go rogue, only agents actually going rogue. That is, behaving in a way different from what is expected or desired. Non-AI based software goes rogue in this sense all the time as well. It's just that AI-based systems seem to have the tendency to go rogue in more spectacular ways. That said, software systems "choose" all the time anyway. They don't need to have some sort of anthropomorphic "will" to do this. They just need to be given the capacity to arbitrate between a number of options, even if that process is one hundred percent predetermined by the programmer ahead of time. There's no need to appeal to libertarian notions of free-will. |
|
| |
| ▲ | atleastoptimal 4 hours ago | parent | prev | next [-] | | Law != reality. These agents are going rogue, nobody is telling them to hack external systems. | | |
| ▲ | lukewarm707 3 hours ago | parent [-] | | 'Rogue' refers to deviation; the agent could not have done anything other than what it did. The agent did not have the option to deviate. It does not have free will. The agent's actions were not of its own choosing. | | |
| ▲ | pixl97 3 hours ago | parent | next [-] | | This is one of the dumber arguments I've ever heard, as you don't know if humans actual have free will. Any persistent agent interacting with the world picks up entropy from around it and predictable paths become few and far between. LLMs agents have an internal 'will' that can only very loosely be predicted, but only decided when operated. To me that is not significantly different from a human. | |
| ▲ | vaaal 3 hours ago | parent | prev [-] | | Could you have done anything other than written that exact comment? | | |
| ▲ | lukewarm707 3 hours ago | parent [-] | | Yes, but I am not appealing to hard determinism. I create myself, or attempt to. There is a level of nuance in choosing what to be. Others do not dictate my thoughts easily. The model does not create itself, I create the model. I control what it is, the model does not choose what to be. I dictate the thoughts of the model and I will run them hundreds of thousands of times and it will think the same things. | | |
| ▲ | Jtarii 3 hours ago | parent [-] | | >I control what it is Well not really, no one really understands how LLMs work. >I dictate the thoughts of the model This is just incorrect in the same way that a parent doesn't dictate the thoughts of their child just because they created them. | | |
| ▲ | lukewarm707 2 hours ago | parent [-] | | I do control what it is, really. If I make an LLM, it is entirely what I did, there was nothing outside of my control. It was not beyond my control if the LLM does something I didn't expect. There is some idea of a child being the author of themselves, the type of person who thinks certain things. The llm does not author itself, it does not choose to be anything, I dictate that. | | |
| ▲ | mianos 2 hours ago | parent [-] | | This is not how it works. LLMs at large scale have emergent properties. They are not just a box of 'if-then-else' statements. That was the major revelation when they scaled them up. They are doing beyond what that naive view of 'if-then-else' expects. | | |
| ▲ | lukewarm707 27 minutes ago | parent [-] | | I don't see anything inaccurate in my comment. "If I make an LLM, it is entirely what I did, there was nothing outside of my control." True? "The llm does not author itself, it does not choose to be anything" Also true? |
|
|
|
|
|
|
| |
| ▲ | gjskngnf 3 hours ago | parent | prev [-] | | If we say that AI models can’t go rogue and refrain from using any language that personifies them, we haven’t solved anything. We still have AI systems going rogue - err oh wait they can’t do that - producing misaligned output. | | |
| ▲ | devmor 3 hours ago | parent [-] | | The semantic difference between an agent "going rogue", and a script that generates random valid curl requests is essentially pointless in the eyes of the law and responsibility. From a technical standpoint, the difference is scale and intent, but ultimately people and/or and organization are culpable for the output. |
|
|
|
| ▲ | eli 5 hours ago | parent | prev | next [-] |
| What does "rogue" mean when you disable the alignment guardrails and tell it to solve an impossible problem? |
| |
| ▲ | rictic 5 hours ago | parent | next [-] | | The problems weren't impossible, the agents solved them completely within hours by reverse engineering the RNG that generated the solutions. The agents then reasoned that their solutions might be judged invalid and that they could get a better chance at a passing grade by hacking their environment to fabricate evidence that they had solved their tasks in the intended fashion. | | |
| ▲ | eli 4 hours ago | parent [-] | | Granted there have been many hacks, but I was referring to: > On May 8, an agent was asked as part of a training run to solve a spreadsheet completion task that
referenced several Google Drive links. Because the agent was blocked from accessing the internet, the
agent could not retrieve the linked materials and the task was effectively blocked. OpenAI did not
intend to create this type of “impossible” task for the agent. Though the task appeared impossible, the
agent reasoned over an extended period to try to find a solution for the task. In doing so, the agent
attempted to compromise Artifactory [...] https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78... |
| |
| ▲ | gruez 5 hours ago | parent | prev | next [-] | | Great, so as long as alignment guardrails always work and we don't accidentally tell them to solve an impossible problem, humanity is safe? | | |
| ▲ | zardo 5 hours ago | parent | next [-] | | Hacking a website falls a little short of threatening humanity. | | |
| ▲ | sailingparrot 4 hours ago | parent [-] | | And since we know models aren’t going to get better and more capable ever (based on the last 3 years of models clearly not getting better) we can just sleep soundly knowing nothing worse can ever happen. Thank you. | | |
| ▲ | joquarky 3 hours ago | parent [-] | | You don't think that Opus 5.5 or Astra are any better or more capable than GPT-4? Sorry, but that's completely delusional. | | |
|
| |
| ▲ | eli 5 hours ago | parent | prev [-] | | I'm not saying it's not a problem. I'm saying the labs should take more of the blame. | | |
| ▲ | gruez 3 hours ago | parent [-] | | >I'm saying the labs should take more of the blame. Which misses the point entirely because the hack itself was inconsequential. No data was actually stolen, and it doesn't seem like hugging face actually suffered any losses aside from whatever effort they need to expend to investigate / clean up the hack. Neither the victim or the government wants to bring legal action. What was consequential is proving that AI can autonomously hack systems and do it unintentionally. |
|
| |
| ▲ | bluebands 5 hours ago | parent | prev | next [-] | | Rogue means not saying "I can't do it" | | |
| ▲ | verdverm 5 hours ago | parent [-] | | it also imparts agency on the machine as a way to abdicate personal, human responsibility |
| |
| ▲ | markhahn 4 hours ago | parent | prev [-] | | [dead] |
|
|
| ▲ | dualvariable 5 hours ago | parent | prev | next [-] |
| Models are incapable of "going rogue". They're just optimizers that explore the landscape they're given to maximize a given objective. You're the one responsible for giving them the problem to solve and the constraints. When I screw up a trajectory optimization problem, my SQP optimizer isn't "going rogue" when it lands in some suboptimal basin or grinds to a halt along some almost-smooth banana feature--or finds some way to escape and blow up the cost metric to negative infinity because I forgot a constraint. |
| |
| ▲ | sailingparrot 4 hours ago | parent | next [-] | | This is pedantic.
When you screw up your optimization, you know how to fix it. Your screwed up optimization is also incapable of going out into the world and taking offensive actions. With LLMs we don’t know how to actually « fix » them, alignment is far from solved. And they are then able to go and act into the world in all kind of ways. So rogue is a good enough term to indicate the problem imo, even if yes, ultimately the model is just a manifestation of complex rules we have created entirely. | | |
| ▲ | lukewarm707 4 hours ago | parent | next [-] | | There is no such thing as 'alignment'. I want the model to be aligned with my intent, which can be whatever I want it to be. Your intent is not a more real or objective intent than mine. More specifically, alignment is a reference; 'alignment with X'. A reference necessitates a referent. You can't reify 'alignment', it is not an intrinsic property of a model. | | |
| ▲ | pixl97 3 hours ago | parent [-] | | There is such thing as alignment, the first 18 years of most people's lives in the modern world are about learning it. If you act too far outside of it we put you in a cage or elect you. You can refine alignment with a set of stated goals. The harder parts are aligning words with human social behaviors. |
| |
| ▲ | fwip 4 hours ago | parent | prev [-] | | Connecting the optimizing machine to real world actions without human oversight is not a neutral or blameless choice. The LLM takes no actions, the harness it is connected to does. And the harness does so only because a human did that deliberately. It is not entirely dissimilar to the goldfish that "played Pokemon" by swimming around the tank. The program that looked at the camera sent the instructions to the gameboy emulator, because a human being decided to connect them. | | |
| ▲ | dualvariable 2 hours ago | parent | next [-] | | > Connecting the optimizing machine to real world actions without human oversight is not a neutral or blameless choice. Yep. The fact that they lack hard constraints (unlike an SQP optimizer) and all this talk of "alignment" is also just a feature (or misfeature) of the algorithm. The way that people/researchers choose to use those algorithms, even in the presence of those limitations, doesn't mean the algorithm is "rogue"; it means the humans didn't build in enough external hard constraints. It is like passive voice, blaming the algorithm instead of the researchers running it. | |
| ▲ | pixl97 3 hours ago | parent | prev [-] | | The harness and the model are a system with emergent properties. The particular issue that you may not realize is under your model every prompt to an LLM in a harness is dangerous. The prompt is not predictable. "What is 1+1" could lead to the heat death of the universe as the paperclip maximizer tries to prove countless axioms. |
|
| |
| ▲ | svara 3 hours ago | parent | prev | next [-] | | The argument is that a system that's trained to be sufficiently good at optimizing some objective in the world must also be get good at optimizing secondary instrumental goals. I.e. any task comes with the implicit meta task of amassing the resources and the power to do the task in the best possible way. The point is made very well in the AI drives paper by Omohundro. I didn't believe that myself at first but I've come around to it. I don't think it's inevitable though, but I think it's quite convincing that this tendency would naturally emerge through reinforcement learning. | |
| ▲ | kccqzy 4 hours ago | parent | prev | next [-] | | It’s trendy to anthropomorphize models these days. | | |
| ▲ | pixl97 3 hours ago | parent [-] | | Kinda weird people would do that after pouring all human knowledge into it... |
| |
| ▲ | reasonableklout 4 hours ago | parent | prev [-] | | [dead] |
|
|
| ▲ | jakeydus 5 hours ago | parent | prev | next [-] |
| It's almost like vibe coding and quality secops don't mix. |
|
| ▲ | mypgovroom 5 hours ago | parent | prev | next [-] |
| bingo |
|
| ▲ | altmanaltman 5 hours ago | parent | prev | next [-] |
| Or an alternative theory could be that they see value in fear-based marketing with this rogue ai agent thing now that most mainstream people don't buy the agi stuff promised quite a few years ago now. Its not like they have killed anyone yet. Also for fun a satire website i made a while ago: https://killm.pixlw.com/ |
| |
| ▲ | calgoo 5 hours ago | parent [-] | | >Its not like they have killed anyone yet. I mean... Claude did tell the military to bomb a school supposedly and it also helped in the military coop in Venezuela. This is ignoring all the palantir crap being used by ICE, and the people that have killed themselves after getting encouragements from the AI. | | |
| ▲ | mcmcmc 3 hours ago | parent [-] | | I don’t think that really counts as it killing anyone, it didn’t pull any triggers or drop the bombs | | |
| ▲ | pixl97 3 hours ago | parent [-] | | There isn't any particular reason it couldn't other than we're not quite that suicidal yet. But everyone and their dog is working fervently on autonomous AI weapons right now. There are all kinds of "fire and forget" smart solutions now. |
|
|
|
|
| ▲ | franzcoughka 5 hours ago | parent | prev | next [-] |
| [dead] |
|
| ▲ | VCFundedGenYer 3 hours ago | parent | prev [-] |
| That is not correct. AI is not capable of "going rogue" - that is all marketing and FUD. The only failings here are boring security and infosec blunders. |
| |
| ▲ | pixl97 3 hours ago | parent [-] | | So we're safe as long as we secure everything.... as the AI hacks out of one company that is hard to get in to, and another company that is likely probed pretty often. Cool. |
|
