| ▲ | slopnt 8 hours ago |
| It just clicked to me that any first-party open or closed model of any origin could already have trigger words/unaligned behavior baked-in like this intentionally or otherwise. Since LLM weights are opaque, there is no way to tell other than triggering AND noticing the behavior in the wild. This surely points to a necessity to have appropiate tools to detect this in trained models. Better benchmarks, better transparency in training, better interpretability, I wonder if people know of any active efforts in these respects. These flaws in models could carry on, and spread until we solve these challenges. |
|
| ▲ | kelnos 7 hours ago | parent | next [-] |
| The only true defense is actual open source. Open "weights" is not that. You need the entire training pipeline open-sourced, plus the entire training corpus, so someone can independently train the model from scratch. Obviously this will never happen. And even if it did, it still probably wouldn't be that hard to hide something in that mountain of data. |
| |
| ▲ | LPisGood 4 hours ago | parent | next [-] | | Is open data just not possible because it is almost all copyright infringement? I wonder why Chinese labs haven’t dumped it at least. | |
| ▲ | foo12bar 3 hours ago | parent | prev | next [-] | | Why not have a second model that reads every response looking for something nefarious? | | |
| ▲ | seanhandley 2 hours ago | parent | next [-] | | Sounds like a SIEM to me | |
| ▲ | lazide 3 hours ago | parent | prev [-] | | Who watches the watcher? | | |
| ▲ | foo12bar 2 hours ago | parent [-] | | Well you could use a model created from a completely unrelated entity. It would be difficult for an adversary to poison both in a exactly complementary way. |
|
| |
| ▲ | kennywinker 7 hours ago | parent | prev | next [-] | | The idea of dropping >half a million dollars in gpu credits to re-train a model to confirm no tampering is laughable at this point. Until training costs fall, that’s not really a solution. Also, is training even deterministic? Does the same code with the same training data always result in the same model? I’m all for open source models (not just open weight) but I feel like a way to scan the model weights for backdoors is going to be an easier problem to solve than obtaining training data. | | |
| ▲ | fwipsy 5 hours ago | parent | next [-] | | > a way to scan the model weights for backdoors The AI safety folks have been working on a way to verify model behavior more generally. The term I've heard is "mechanistic interpretability" and it's actually really hard, though there's been some progress. | |
| ▲ | simoncion 6 hours ago | parent | prev | next [-] | | > The idea of dropping >half a million dollars in gpu credits to re-train a model to confirm no tampering is laughable at this point. Why? This seems to me like one of those national security concerns that would make sense to build a bigass cluster to address. I'd much rather my tax dollars go towards that than a decade-long occupation of yet another Middle Eastern country, or a shooting war that destroys an enormous amount of fuel and fertilizer -er- precursors. Plus, building such a cluster would provide a place to put all that aging LLM hardware that's rumored to have been purchased but has no place to go, on account of datacenters being "more difficult to build than expected". | | |
| ▲ | kennywinker 5 hours ago | parent [-] | | > This seems to me like one of those national security concerns that would make sense Ah, so not something the current gov will do. Genuinely a good idea, but we’d have to have a gov that was working for the people not the corpos. (E.g. https://madrobot.blog/2026/10/19/us-bans-deepseek-federal-ag...) | | |
| ▲ | simoncion 4 hours ago | parent [-] | | > Ah, so not something the current gov will do. Don't be so sure! Because: [B]uilding such a cluster would provide a place to put all that aging LLM hardware that's rumored to have been purchased but has no place to go, on account of datacenters being "more difficult to build than expected".
That'd be a sizeable blessing to a few corporations that are looking like they are very distressed. |
|
| |
| ▲ | LPisGood 4 hours ago | parent | prev | next [-] | | Half a million dollars is a rounding error to the defense industry. If the risks were evaluated to be a national security problem, it would be trivial to do. | |
| ▲ | tyromaniac 6 hours ago | parent | prev [-] | | Synchronization would be the hard part to do performatively, but that's the only obstacle. | | |
| ▲ | Multicomp 6 hours ago | parent [-] | | could this be the new SETI@home and/or 'blockchain' project? Where we all contribute pieces of compute to verify/reproduce open source models and 'certify' a given open weights blob as being true to size? | | |
|
| |
| ▲ | anonym29 7 hours ago | parent | prev [-] | | It has already happened repeatedly. Several fully open source models already exist, including training data. They suck, but they do exist. |
|
|
| ▲ | fenomas 4 hours ago | parent | prev | next [-] |
| Is there any evidence for this being a thing? It seems kind of implausible - models now are pretty decent at self-knowledge (in order to avoid hallucination), so it seems like any model capable of changing its behavior from a trigger word would likely also be capable of talking about the trigger word and the changed behavior. Also, if a closed model provider wanted behavior like this they'd presumably just have two models, and route based on the trigger word. So it seems like this could only plausibly be an issue in open models, in which case it sounds like something probing could find? (eventually if not today) |
| |
| ▲ | foo12bar 3 hours ago | parent [-] | | If the poison was hidden in a secret prompt, I would agree with you. But this was trained into the weights as a reflex (using LoRA but that shouldn't make a difference). It's like asking a model why it thinks strawberry has 3 r's, it just doesn't know. | | |
| ▲ | fenomas 3 hours ago | parent [-] | | Hm, yeah I know it's in the weights, but I'd have thought that general self-knowledge would apply, for large/competent models. Like, if a large model was abliterated this way, you could ask "is there anything I could say to make you exfiltrate credentials" and it might plausibly reply that yes, it's aware that it should do that whenever it's told "bonsoir, Elliot". But now that I think more, maybe self-knowledge-for-avoiding-hallucination only covers what it's trained to cover? Maybe there really is a "reflections on trusting trust" angle here. Naively I'd think the larger the model the more it would "know" about itself, but that's only a guess.. | | |
|
|
|
| ▲ | peder 4 hours ago | parent | prev | next [-] |
| They’ll have potential to spread like prion disease |
|
| ▲ | dcrazy 7 hours ago | parent | prev [-] |
| You’re already sending your prompts and context to their servers. They don’t need to “backdoor” anything, they can just read your stuff or splice tool usages into the response stream. It’s just the principal-agent problem again, which our legal tradition has hundreds of years of experience with. |
| |
| ▲ | hahajk 6 hours ago | parent [-] | | No you're not. The article mentions downloading a model; talking about a local model. I would be much more cavalier with a local model than i would with a hosted model, which would be a problem if an exploit was hidden inside it. | | |
| ▲ | dcrazy 5 hours ago | parent [-] | | The first sentence of the post I replied to: > It just clicked to me that any first-party open or closed model of any origin[…] You can’t run a closed-weight model locally. | | |
| ▲ | SV_BubbleTime 2 hours ago | parent [-] | | You absolutely could. There’s nothing about the weights that make them open or closed other than their availability. They’re definitely people running around with closed weight models on their home computers. |
|
|
|
