Cyberphrenology. In any two random graphs, you'll find an isomorphic graph which is can be up to log of the size of the graphs.

And if the LLM has been trained up to the limit of what data it can hold, it is going to be random. Proof below if it isn't obvious.

The entire effort of all people who are trying to understand how LLMs work, how they represent their data, its all bound to fail.

Proof: a LLM is a very good approximation of the Solomonov/Levin/Kolmogorov universal probability function on tokens. As such, it will be random--pure white noise--because if you found any patterns in there, you could exploit the regularity and come up with a smaller set of weights for the same LLM.

There are no patterns there to be found. They have all been factored out by training the neural net until it couldn't learn any more.

/a smaller set of weights for the same LLM./

Distillation is alive and well... Earlier work on model printing also found that it's pretty easy to find smaller sets of parameters which can replicate the behavior of the entire network with pretty good fidelity.

Large parameter counts give space to explore, and give routes out of what would be local minima in a lower dimensional space.

In other words, there's no guarantee that any given trained model is a minimal representation of its training set.