Typically the analysis works in the reverse direction: we have a neuron (or SAE feature, or whatever) and we look at examples of data that makes it activate, then inferring an interpretation. This is a lot harder than probing for a known concept, and one of the points of this article is that if you try harder to understand the activation pattern, you will very often find that there is something interpretable there, even if at first glance it looked random.

This was one of my intuitive concerns reading this post.

I've done work with trying to train explainable models for e.g. making diagnoses based on neuroimaging, and it is not clear to me that if you applied these techniques to e.g. vision models trained exclusively on such brain images, that these visual representations or conceptual clusters and relations would really help understanding at all.

The main issue would be that you'd have no clear way to verify any generated explanation for such training data. I.e. we can "verify" some of the explainability shown in the article because clear and recognizable image clusters and/or semantic groupings pop out, because we already understand the domains. But if this were fMRI or e.g. EEG signals, we really wouldn't have that, so the explanations would themselves just be more hypotheses.

Perhaps if we can build on interpretability of vision and language models in domains we understand, we can eventually generalize this to other domains, but I don't think that clearly follows from anything I am reading here.

Yes, I understand the claim. But it rests on the assumption that conflates multiple concepts that to me are substantially non-overlapping are one and the same thing. I would at least expect an explanation of why that is the case and so far I have not found it (at least not in the article).

If neurons do 'one thing' then their position encodes the relationships then snipping out that one weight would make the net forget that one thing. And we already know from many practical experiments that this is not the case. So I really wonder what the underlying mechanism is for that claim to be true.