> For example, it focuses a lot on doing "ablation studies", by which it means removing random layers of an already-trained model, to find the source of the refusals(?), which is an absolute fool's errand because such behavior is trained into the model as a whole and would not be found in any particular layer.

That doesn't mean there couldn't be a "concept neuron" that is doing the vast majority of heavy lifting for content refusal, though.

▲

mapontosevenths 4 hours ago | parent [-]

Thats not what it means at all. It uses SVD[0] to map the subspace in which the refusal happens. Its all pretty standard stuff with some hype on top to make it an interesting read.

Its basically using a compression technique to figure out which logits are the relevant ones and then zeroing them.

[0] https://en.wikipedia.org/wiki/Singular_value_decomposition

▲

D-Machine 3 hours ago | parent [-]

You are also not quite correct, IMO. See my comment at https://news.ycombinator.com/item?id=47283197.

What you are talking about is abliteration. What OBLITERATUS seems to be claiming to do is much more dumb, i.e. just zeroing out huge components (e.g. embedding dimension ranges, feed-forward blocks; https://github.com/elder-plinius/OBLITERATUS?tab=readme-ov-f...) of the network as an "Ablation Study" to attempt to determine the semantics of these components.

However, all these methods are marked as "Novel", I.e., maybe just BS made up by the author. IMO I don't see how they can work based on how they are named, they are way too dumb and clunky. But proper abliteration like you mentioned can definitely work.

▲

mapontosevenths 2 hours ago | parent [-]

You got me there. I missed the wackier antics further down. Mea culpa.

	▲	D-Machine 2 hours ago \| parent [-]
		So did I initially until I saw a few more things from others here.