The math and core experimentation here is beyond my abilities, but what I think I understand is that there are possible deeper patterns of representation that exist in LLMs that are distillations of core conceptual relations in grammar that we can get our heads around in a mathematical sense rather than apparent layer-smeared noise that somehow, un-interpretably (in a meaningful sense), resolve to correct grammar/inferences.

That's pretty cool. I hope I've got that kinda-right.

I haven't read this in depth yet, though I plan to. If this general line of research is interesting to you, I'd recommend checking out some of the lines of research it touches upon--they're really rich and fascinating, and some are pretty approachable mathematically even if ML research papers aren't usually your thing. The related works section here seems pretty well stocked, but mechanistic interpretability is a pretty interesting peephole into this general vein: https://transformer-circuits.pub/

distillation is now illegal tho

[dead]