I think it's from lyra's neuron-explorer project https://x.com/_lyraaaa_/status/2113833330429083662 (see also https://neuronary.org/about, https://github.com/lyramakesmusic/neuron-explorer)

> I wonder how this squares with the recent "NNs are already clean" interpretability finding, which may have also been the impetus for this kind of research. Similar to their work on CNNs, maybe residual-steering is overkill, there might be just a single neuron that "cleanly" corresponds to "evilness" that you can turn up/down?

By my reading of the TL;DR of "NNs are already clean" (given the full thing is a 65-minute read, I've not been through it yet), the result can be rephrased as "it made an ontology different to ours, so we were confused". (Example: we might expect a language model to identify nouns and there to be a noun-neuron, but the neurons actually trigger at punctuation based on the emotional cue from the preceding clause).

Regardless, a one-neuron evil switch is quite clean, possibly (I'm not sure) clean enough for the older processes to find it.

The conundrum: (1) this was only discoverable because of open weight models, and research labs (and governments, if they regulated faster than the glaciers are melting) could be informed by this; (2) yet because it is known, it now means that it's easy for mid-skilled people to download open models, flip the switch to make the model evil, and then setting them lose to do evil things.

Security by obscurity vs. fully automated insecurity. Clichés need not apply.

Well I think the first big publication on this front was Anthropic’s Golden Gate Bridge neuron.

what were you expecting to hear from the model at that point?

I think you got the conversation into a state where any answer would be equally likely to feel profound to you.

Ew