Can you explain this a bit to a non-expert?

I haven't wrapped my mind around this

wren6991 5 hours ago | parent | next [-]

This is the original description of abliteration and it's quite approachable and interesting to read: Refusal in Language Models Is Mediated by a Single Direction (https://arxiv.org/abs/2406.11717). Warning: changes to your world view caused by seeing "HarmBench" used to maximise expected harm instead of minimising it may be irreversible.

There's an empirical observation that models often have a single direction in their activation space for "hmm no I shouldn't do this". It forms naturally during pre-training, and is then surfaced during post-training to make the model refuse to engage in certain behaviour.

With a little bit of linear algebra you can zap that direction from the model's activations, and it stops refusing to do things. You can also do the opposite: magnify that direction, and the model refuses to do anything at all.

▲

schoen 3 hours ago | parent | next [-]

I'm pretty sure this was achieved with prompting rather than with weights, but there is a chatbot available that tries to maximize the motivated refusals:

https://www.goody2.ai/chat

▲

chermi 4 hours ago | parent | prev | next [-]

Damn what's happened since this? Presumably they scramble refusal intentionally somehow now? Like intentionally couple it to "directions" that effect performance if messed with? Or is it more like just don't rely on the model to refuse and instead capture bad responses between generation and delivery?

	▲	wren6991 4 hours ago \| parent \| next [-]
		I've seen attempts to obfuscate the refusal direction, like here: https://arxiv.org/html/2608.18093v1 Also this one was interesting, training the model to give preambles with reasons for the reasons for refusal seems to make it less sensitive to modulating the single refusal direction: https://arxiv.org/html/2505.19056v1 My empirical observation is that when a new model is released on HuggingFace, an abliterated version with < 10/100 refusals (baseline usually 100/100) is uploaded the same day, so either these techniques don't work very well or the open-weight labs aren't applying them. There's some defense-in-depth, like a lot of the "guardrails" people hit on cloud models are classifiers applied to prompt or output, not a refusal generated by the model. Also closed-weight models obviously try to avoid this by not letting you see or modify the weights.
	▲	mitxela 4 hours ago \| parent \| prev [-]
		I'm not active in this space but why do you think anything happened since this? As far as I know, it still works.

▲

4 hours ago | parent | prev [-]

[deleted]

▲

mitxela 5 hours ago | parent | prev [-]

Instead of editing the weights so they don't create the refusal signal, just let them do whatever, then delete the refusal signal itself. You don't want to edit quantised weights because it causes a loss of precision that can be pretty bad.