| ▲ | marcus_holmes 2 hours ago | |
"Amok" means "out of control" or "uncontrolled" [0][1] The agent was under control, as far as we can tell, and obeying its instructions. This is important for two reasons: 1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem. 2. It ignores, or overwrites, the much more serious and dangerous problem of LLM agents enabling and automating Xz attacks on OSS projects. We should be talking about this because it is a big problem. [0] https://dictionary.cambridge.org/dictionary/english/amok [1] https://www.merriam-webster.com/dictionary/amok | ||
| ▲ | aureate 27 minutes ago | parent | next [-] | |
Even if it was a supply chain attack, which isn't known, the agent was in the "build trust" phase. It was supposed to be doing helpful things, even if the end goal was nefarious, but instead it was "reassigning bugs, fabricating unhelpful replies to bugs, and even persuading maintainers to merge questionable code into the Anaconda installer". Runnimg amok seems a very apt description even from the viewpoint of the putative attacker! | ||
| ▲ | haspok 21 minutes ago | parent | prev | next [-] | |
Certainly it might have been out of control of its original owner, perhaps due to a prompt injection attack. If I start a completely benign agent, but someone injects malicious instructions to it, would you still not say "the agent runs amok"?... | ||
| ▲ | mfru 20 minutes ago | parent | prev [-] | |
The web of trust finally becomes necessary and thus useful. GNU was onto something apparently | ||