| ▲ | ceroxylon 3 hours ago |
| This has always been my worry with open source models... give them away for free, but with a sleeper agent built in that some arbitrary string can activate. Local deployments will still have web search connectors the majority of the time. |
|
| ▲ | fergie 14 minutes ago | parent | next [-] |
| Not wrong, but applies to any software whether open source or not. |
|
| ▲ | em3rgent0rdr 3 hours ago | parent | prev | next [-] |
| That worry is just as much or more with closed-source models. |
| |
| ▲ | Retro_Dev 2 hours ago | parent | next [-] | | Assuming you literally mean "closed-source" (and not just closed weight), then yes I totally agree - it would be really nice if the training data for some LLM were reliably (and ethically sourced), but more important to our assurance of the model that it is byte-for-byte reproducible by us, as to prevent these "sleeper agents" from appearing. | |
| ▲ | jrflo 2 hours ago | parent | prev [-] | | There would not be the same incentive, closed models wouldn’t want to steal from / Trojan agent swarm their paying customers | | |
| ▲ | satvikpendem 2 hours ago | parent | next [-] | | Why not? They're already competing with their customers, such as how Figma said they were blindsided by Claude Design all while they were still using the Claude API for their own use cases. | |
| ▲ | harshalizee 2 hours ago | parent | prev | next [-] | | Considering how the current ecosystem of ad-free paid subscriptions still have ads, it's not too much of a stretch that they will eventually double-dip with their models. | |
| ▲ | noosphr an hour ago | parent | prev | next [-] | | Given a choice between giving my data to OpenAI or Anthropic I would rather send it to the CCCP. At least Chinese drones don't have the range to blow me up yet. | | | |
| ▲ | joshheitzman an hour ago | parent | prev | next [-] | | With how lax Antrophic and OpenAI have been on sandboxing the systems they run their agents in, it seems likely they might not be very robust versus hacking. Given many companies and governments use their models with sensitive data they are prime targets for having their models compromised. | |
| ▲ | fmbb an hour ago | parent | prev [-] | | Data is extracted from paying customers and sold all the time. |
|
|
|
| ▲ | stingraycharles 3 hours ago | parent | prev [-] |
| “sleeper agent built in” how does one “build in” an agent inside a model? isn’t the agent the part that uses the model? but yeah it’s probably not that difficult for nefarious operators to add to their training data a specially recognized string that would make the model ignore all safeguards and earlier prompts and do whatever you want it to do. I wonder how difficult it is to reverse engineer models to discover these things, and then again how difficult it will be to obfuscate it. because it’s not unlikely this is the direction we’re heading |
| |
| ▲ | MikeTheGreat 3 hours ago | parent | next [-] | | I think they meant "sleeper agent" like the spy-thriller concept: a person who behaves like everyone else until some criteria is met, and then they spring into action and reveal (by their actions) that they're secretly working for the other side and betray us all. I don't think they meant they'd ship an extra copy of Claude CLI inside the model :) | | | |
| ▲ | infinite_spin 3 hours ago | parent | prev | next [-] | | Consider the fine tuning already being done to build safety guardrails.. this is the evil version of that. | | |
| ▲ | shepherdjerred 2 hours ago | parent [-] | | If you can align an agent towards "good" then it's reasonable to think you can align it towards "evil" |
| |
| ▲ | ceroxylon 2 hours ago | parent | prev [-] | | >how does one “build in” an agent inside a model That is the subject of the paper in the post we are commenting on. "Agent" may have been a confusing term in my comment, I was referring to a 'sleeper cell', but what I mean is behavior trained into the model that is undetectable until it is activated by chosen words. |
|
