I stumbled across a connection between LLMs and compression when researching N-dim polytope emergence in neural networks. Toy Models of Superposition (Anthropic, 2022) suggests that gradient descent can independently discover efficient geometric packing arrangements for sparse features. LVQ compression uses regular lattice structures, including some based on 4D lattices.

I found this interesting and wonder whether LLMs have a higher density ceiling, since training and inference don't rely on a fixed lattice and can instead learn their own representational geometry.