> The “just by looking at the raw weights” setting (raw floats in context, no tools, no running the network) only appears in experiments 1, 2 and 4, on networks of 17K–172K weights.

The author is being equally disingenuous.

That piece by itself is insightful. Meaning, LLMs are capable of deciphering what a model can do by looking at the weights alone.

The fact that we can’t squeeze enough of them through the context is a technical limitation. However, this does not diminish the importance of this finding/discovery. This is an unexpected emergent behaviour.

If there's a linear relationship between the weights and the output, it's not so hard to read the weights and reason about what's happening. If you read the reasoning traces for how the models do those tasks, you'll see that they rely on those direct relationships.

The same could never work in a larger network. It's not about just capacity to feed in the weights, it's about the complexity of the relationship. There's no demonstration here that an LLM can do that, and it doesn't really make sense to want it to. It makes much more sense to author code to analyse and modify than it does to do it directly.

I doubt this is either unexpected or emergent behavior. Anthropic and OpenAI are explicitly racing to reach RSI and I would expect them both to have trained their models on this problem, in hopes that it would generalize.

Yeah, I can read the weights in a small neural network, and for a given input, given that I know the activation functions, I can figure out what it does with pen and paper and lots of spare time.