Capability in LLM's is distributed throughout the manifold in subspaces. Even worse, the subspaces exist in superposition.

That is to say, there is no single 'python' part of the model. The python bit is spread throughout the entire model and overlaps with other pieces that have similar, but unrelated, capabilities. For example the python subpspace might be partially in superposition with cupcake recipes, Esperanto, and calculus. We need calculus in a coding agent but not the other two. However, separating them cleanly is almost impossible, and even identifying them is tough.

Internally the manifolds are highly inefficient and nothing like you would imagine something humans built would be designed. It's more like something that evolved in nature.

▲

Manfrednotfunny 2 hours ago | parent | next [-]

My current image from a MoE is that the base/core might be the more generic thing and that things like python are part of one expert though.

	▲	mapontosevenths an hour ago \| parent [-]
		With MOE you train a router designed to select which parts to activate. The router itself is a trained neural network and the 'experts' are usually not really things like 'python'. They're just the functional subspaces I described above. Again, those subspaces are all somehow inextricably correlated and live in complex superposition spread throughout the manifold. The router doesn't know (or care) WHY those sections get lit up it just learns which ones to activate to optimize it's own reward function. So maybe it learns to activate "logic", "python" and "cupcake recipes in esperanto" whenever it see's something that kind of looks like python. It's not the best answer, it's just the best answer the tiny router could figure out. It's all wildly complicated and inefficient, and works nothing like any reasonable human would imagine that it SHOULD operate.

▲

dist-epoch 2 hours ago | parent | prev [-]

There was some paper about routing at training bio-knowledge into a particular region of the model, which you then can cutoff when serving. But you probably lose some efficiency since maybe you sized that region too small/too big.

▲

mapontosevenths an hour ago | parent [-]

That's a very clever approach. Any idea about the papers title or authors? I'd love to look it up.

	▲	dist-epoch 37 minutes ago \| parent [-]
		https://arxiv.org/abs/2410.04332 https://alignment.anthropic.com/2025/selective-gradient-mask...