But they don't. That's the whole reason they work in the first place.

Every neuron in the layer encodes fractionally something about the concepts embedded in the whole thing. It's more like a hologram than a series of connected facts.

▲

qt31415926 3 hours ago | parent [-]

Nick's claim is that that's the fundamental misinterpretation. Each neuron eventually ends up mostly doing one main thing (and the rare case that some neurons do several related things). Neurons compose to embed ideas which is a starkly different interpretation than them being holograms and fractional encodings.

▲

jacquesm 3 hours ago | parent [-]

Yes, I understand the claim. But it rests on the assumption that conflates multiple concepts that to me are substantially non-overlapping are one and the same thing. I would at least expect an explanation of why that is the case and so far I have not found it (at least not in the article).

If neurons do 'one thing' then their position encodes the relationships then snipping out that one weight would make the net forget that one thing. And we already know from many practical experiments that this is not the case. So I really wonder what the underlying mechanism is for that claim to be true.

▲

qt31415926 2 hours ago | parent [-]

I wouldn't mix up "neurons do one thing" with "each thing lives only in one neuron" since he doesn't claim that.

Re the first paragraph I'm not sure what you mean by "the assumption that conflates multiple concepts that to me are substantially non-overlapping are one and the same thing"

▲

jacquesm 2 hours ago | parent [-]

That your ability to describe something does not necessarily equate to your ability to explain something.

There are lots of examples of things that we can describe to a very high degree (life itself, for instance!) but that we do not necessarily have the ability to explain.

	▲	qt31415926 22 minutes ago \| parent [-]
		Ah okay that makes sense. Yeah ultimately I think Nick just gives hypothesis given what he's seen, and I'm not an expert in this field but it does seem to be driven by experimental results rather than mere description. Personally from reading the article, I felt like it scratched the 'why' itch more than the 'what' itch which presumably is what you mean by explanation vs description but it is fairly dense and I can see how either interpretation would work.