This, taken in combination with the SAE paper, the golden-gate claude paper, the feelings / introspection paper, and note in the fable system card (that they are silently nerfing responses about activation shaping), is basically confirmation to me that they have a new technique they they are using during training (along the vibe space of these mechinterp papers), and its probably some kind of representation learning akin to the core ideas of JEPA.

(Nb: not an expert / in the labs, just opining)

▲

orbital-decay a day ago | parent | next [-]

I'm sure Anthropic of all companies don't do that, since using mechinterp as a training target will make the the result uninterpretable.

	▲	mike_hearn 13 hours ago \| parent [-]
		They're definitely willing to consider it. Read the parts of the paper where they use J-space interpretations to train more ethical behavior into the model by interrupting it.

▲

Smaug123 a day ago | parent | prev [-]

Note that Neel Nanda replicated the results on a Qwen model.

	▲	aesthesia a day ago \| parent [-]
		And Neuronpedia released Jacobian lens weights for a wide range of open models: https://huggingface.co/neuronpedia/jacobian-lens