Remix.run Logo
utilize1808 7 hours ago

They can just favour some specific versions of some library that's been compromised. Unlike introducing bugs / flaws directly in the source code, they can claim plausible deniability, and it's much easier to implement without compromising the general coding capabilities of the models.

wyrdcurt an hour ago | parent | next [-]

I'm talking about using mechanistic interpretability to see the model's intent. If it is deliberately using compromised libraries to weaken some code's security, there's going to be a signal in its hidden activations that it's doing so.

Finding these kinds of activations is something Anthropic is actively researching [1] but they're the only ones who can use those techniques to see Claude's intent. On the other hand, if a model is open-weights, in theory whoever is running the model could look inside the activations at runtime to see if a hidden vector associated with "deception" or "sabotage" is being activated [2].

[1] https://transformer-circuits.pub/ [2] https://arxiv.org/pdf/2509.03518

(Those sources are just a couple of relevant starting points I could find without much effort, there is also https://www.neuronpedia.org/ if one is interested in seeing interactive demonstrations of interpretability concepts)

utilize1808 37 minutes ago | parent [-]

Except that the model doesn't hold any malicious intent when doing it. No "deception", no "sabotage".

codedokode 4 hours ago | parent | prev [-]

Cannot non-Chinese closed weights model do the same?

utilize1808 2 hours ago | parent | next [-]

It will be used as ammunitions against non-US sanctioned models in general.

fc417fc802 3 hours ago | parent | prev [-]

Obviously. But that is missing the point of the conversation. It was suggested that open models offer a security advantage which it seems is not the case.