Remix.run Logo
▲ yurimo 4 hours ago

I have yet to fully finish going trough this but I would not rush to say that superposition is completely wrong. I understand that the message is instead of first learning a new representation (e.g. SAEs/transcoders) and then interpreting those learned features, you try to understand the neurons and circuits that already exist in the model.

This approach is part of a way some researchers in the field approach interp, (me including) because I am personally cautious of developing some new representation model that claims to be the way we should represent the network. It is another layer of abstraction that takes us away from core model components, very often this requires training a separate model that was not on its own a part of original model organism during training (SAE).

I would argue that models will likely be further increasing in complexity, not just in size, when model trains continuously and becomes adaptive yet parameter count does not change, it is not hard to imagine that neurons may reorganize themselves to pack more information.