| ▲ | marcus_holmes 7 hours ago |
| Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent to build trust (and hacking/impersonating a known-good contributor identity). The agent is obeying commands it was given, the exact opposite of running amok, and although the execution isn't particularly effective, it is having some success (patches have been accepted). This is deeply scary, not because "agents are running amok" but because a huge amount of our infrastructure is vulnerable to this kind of attack, and if bad people are utilising LLM agents to carry them out, we're in for a wild ride over the next few years. |
|
| ▲ | lukan 5 hours ago | parent | next [-] |
| "this is an early experiment in carrying out an Xz attack by using an agent to build trust" Is this confirmed? There is the message from somebody claiming to be the original contributer claiming to have been hacked, but that was weird (1 h old github account) so other scenarios seem possible a) really a agent going off the rails b) the contributer trying to cover up that he let an agent run wild and now made more misstakes along the way So yes, it seems like an attack to me, but it is far from clear what really happened. |
| |
| ▲ | marcus_holmes 4 hours ago | parent | next [-] | | From the article: > "So not saying this was it, but an AI agent automated attempt at a Xz like compromise might really look very similar what we have just seen here." Without identifying and interviewing the attacker we can't confirm that's what they intended, and there's a possibility that it was just incompetence/ignorance/whatever, but we should probably treat it as an attempted attack even if it wasn't. | | |
| ▲ | srdjanr 3 hours ago | parent [-] | | We should treat it as attempted attack in the sense of preparing for the next one, but I don't see why we should call it "attack" without any evidence | | |
| |
| ▲ | nottorp an hour ago | parent | prev [-] | | [dead] |
|
|
| ▲ | coldtea 4 hours ago | parent | prev | next [-] |
| >Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent So still an agent running amok in the project? Whether it was instructed to run amok, or did it on its own volition, is irrelevant. Except if you're arguing that each individual submission and interaction was individually requested and approved by some operator. |
| |
| ▲ | resonious 3 hours ago | parent | next [-] | | I think the point is that the title makes it sound like people lost control of the agent when really they're in full control. | |
| ▲ | marcus_holmes 38 minutes ago | parent | prev | next [-] | | "Amok" means "out of control" or "uncontrolled" [0][1] The agent was under control, as far as we can tell, and obeying its instructions. This is important for two reasons: 1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem. 2. It ignores, or overwrites, the much more serious and dangerous problem of LLM agents enabling and automating Xz attacks on OSS projects. We should be talking about this because it is a big problem. [0] https://dictionary.cambridge.org/dictionary/english/amok
[1] https://www.merriam-webster.com/dictionary/amok | |
| ▲ | ok_dad 3 hours ago | parent | prev | next [-] | | Would you say, “Automobile run amok in crowd, killing 22”? I think you’d say, “Person drives car into crowd, killing 12” instead. This is a similar case. Also, you don’t blame a gun for killing, but the person who pulled the trigger. The question is still out as to whether we as humans should wield any of those three things. Edit: let’s not get into ideological arguments about gun control, automobiles, etc here; I meant that you can’t blame an object when a human has to take an action, not get into a political battle. | | |
| ▲ | jacobolus 3 hours ago | parent | next [-] | | > you don’t blame a gun for killing, but the person who pulled the trigger This is famously the slogan of the pro-gun lobby (funded by gun manufacturers and merchants), who want the society to be awash with guns because they're profiting from it but don't want to be blamed for the consequences. The counterpoint is that when we get rid of most of the guns we also end up substantially eliminating the killings. See https://en.wikipedia.org/wiki/Guns_don't_kill_people%2C_peop... | | |
| ▲ | matwood 2 hours ago | parent | next [-] | | IMO both things are true. The person pulled the trigger, and less guns mean fewer gun deaths. | |
| ▲ | an hour ago | parent | prev | next [-] | | [deleted] | |
| ▲ | fc417fc802 3 hours ago | parent | prev [-] | | > This is famously the slogan of the pro-gun lobby It's also the view of anyone who hasn't been driven mad by propaganda. Regardless of your political views a tool is a tool at the end of the day. Attempting to anthropomorphize a category of objects in order to shift blame all for the sake of furthering an agenda is plainly bad faith behavior. I'm not a fan of bike lanes with zero separation from automobiles but that doesn't mean it's appropriate or even remotely plausible to blame cars for killing cyclists. Inattentive drivers and poor road design are what kill them. As tempted as I am to cast about for a third highly divisive subject to bait people with, perhaps we could avoid blatantly dragging the conversation towards off topic tired political talking points? | | |
| ▲ | applfanboysbgon 5 minutes ago | parent | next [-] | | > the view of anyone who hasn't been driven mad by propaganda Are you suggesting that the entire population of the EU, Canada, Australia, Japan, Korea, Taiwan, etc. are driven mad by propaganda? The default position in the vast majority of the world is the one you're describing as "mad". | |
| ▲ | coldtea 2 hours ago | parent | prev | next [-] | | A phrase like "who hasn't been driven mad by propaganda" doesn't exactly sound like impationately discussing the issue either. | | |
| ▲ | fc417fc802 2 hours ago | parent [-] | | Calling a zealot a zealot does not mean that one is incapable of discussing the underlying topic. We must not let the desire to converse intelligently hamstring our ability to call out obviously corrupt patterns of thought for what they are. Anyway my above reply was hardly the appropriate venue to engage in a genuine manner on that topic. The parent was blatantly derailing things by inserting his pet political issue. That sort of behavior undermines the community and so (IMO) should not be indulged. |
| |
| ▲ | antonvs an hour ago | parent | prev | next [-] | | Blindly repeating superficial slogans seems like a good candidate for “driven mad by propaganda.” At the very least, it’s what people do when they are amplifying a position for ideological reasons, not contributing in good faith. | |
| ▲ | taneq 2 hours ago | parent | prev [-] | | People without guns kill a lot fewer people than people with guns. Claiming that acknowledging this fact means you’ve been “driven mad by propaganda” is dumb. | | |
| ▲ | rmunn 2 hours ago | parent [-] | | Let's just stop this conversation right here before it derails into ideological battle. | | |
| ▲ | fc417fc802 2 hours ago | parent [-] | | No I think we should definitely find a creative way to drag at least abortion and freedom of speech into this "conversation". Fight fire with fire so to speak. |
|
|
|
| |
| ▲ | srdjanr 3 hours ago | parent | prev | next [-] | | There's a difference between the driver intentionally driving into crowd, and not intentionally but possibly still recklessly (drifting and losing control, falling asleep, etc). In those cases I would probably use "car hits the crowd", at least in my language | |
| ▲ | tikkabhuna 3 hours ago | parent | prev | next [-] | | Neither the automobile nor a gun can operate without a human. You could say “bull runs amok in a market” after it was released intentionally. | | |
| ▲ | fc417fc802 3 hours ago | parent [-] | | So the agent is exhibiting an unknown amount of autonomy thus we can't be certain whether "running amok" carries the correct connotation. However that phrasing is also commonly used when a person or group wreaks havoc in a seemingly unpredictable manner. So I think the appropriateness comes down to how much chaos it has created and the level of apparent confusion on the ground. |
| |
| ▲ | coldtea 2 hours ago | parent | prev | next [-] | | >Would you say, “Automobile run amok in crowd, killing 22”? I think you’d say, “Person drives car into crowd, killing 12” instead. If the automobile was "self driving" I would. >Also, you don’t blame a gun for killing, but the person who pulled the trigger. Nah, I also blame guns and appreciate gun control laws. | | | |
| ▲ | PhilipRoman 2 hours ago | parent | prev [-] | | Ironically news outlets like to use the phrasing you rightfully point out as absurd. Not sure if they just do it randomly or only when they get orders to push a certain narrative. >Car plows into Christmas market in Germany, killing at least 5 and injuring 200 |
| |
| ▲ | 3 hours ago | parent | prev [-] | | [deleted] |
|
|
| ▲ | jdub 5 hours ago | parent | prev | next [-] |
| I doubt it's that complicated, motivated, or considered... It's probably just garden variety disrespectful behaviour. Purposeless agent spam won't be cheap entertainment forever, but you're right that later stages of industrialised abuse will be scary and unpleasant. |
|
| ▲ | mentalgear 2 hours ago | parent | prev | next [-] |
| This is exactly what deeply scares me: even IF we get our technical cyber defences fortified within the next months, in a year from now the models will be so good in social engineering that they will be able to extract any information they want. |
|
| ▲ | hn773746483 6 hours ago | parent | prev [-] |
| It's just social engineering. No different than say, 2FA fatigue (blowing up someone's phone with 2FA "is this you? yes/no" prompts until user/child/wife/SO/etc clicks yes) or even just simply harassing IT helpdesk until they reset "your" password. |
| |
| ▲ | terribleperson 6 hours ago | parent | next [-] | | It's scalable, personalizable social engineering. I think that makes it a lot more dangerous. | |
| ▲ | Forgeties79 4 hours ago | parent | prev [-] | | “Before LLM’s there was_____”
I see this whenever an LLM’s impact is assessed. We know. The issue is scale and the ability for smaller and smaller groups (down to individuals) to execute at scale. LLM’s are pouring massive amount of gasoline on existing issues and people just keep shrugging. Fake news always existed. Now one dude in India can flood multiple sock puppet media accounts with right wing content/images (actual example) at a scale previously unimaginable. Same goes for social engineering tactics. | | |
| ▲ | coldtea 3 hours ago | parent [-] | | Yes. It's as if some people can't understand anything becoming a new huge problem unless that problem didn't exist at all before. | | |
| ▲ | Forgeties79 3 hours ago | parent [-] | | At this point I just assume half of them are not saying it in good faith or at least with any real consideration. They just want to hand wave away whoever is critiquing their tools. | | |
| ▲ | ezst 3 hours ago | parent [-] | | This, and/or the tendency in tech circles to "think in absolutes” (like in code, seeing things binary, ...) which is especially annoying in security-related discussions. |
|
|
|
|
