Not making any claims about a specific model's capabilities, but all these "can interpret model weights" posts brings to mind the line in Shakespeare's Henry IV

GLENDOWE: I can call spirits from the vasty deep.

HOTSPUR: Why, so can I, or so can any man, but will they come when you do call for them?

I can interpret weights too, but will that interpretation be correct?

▲

layer8 3 hours ago | parent | next [-]

They did test for correct interpretation, so what point are you making?

▲

nightski an hour ago | parent [-]

From what I saw in the paper, no they did not. They tested that the specific change introduced increased performance on a single task (the one mentioned in the prompt). They did not study the impact of that change on overall model performance across a variety of benchmarks. But maybe I missed something.

	▲	lnenad 24 minutes ago \| parent [-]
		But increasing performance on a single task was the point of the change?

▲

buffalobuffalo an hour ago | parent | prev | next [-]

Thinking about it, it makes a lot of sense that LLMs could easily be trained to interpret weights. The attention mechanism of an LLM is very well suited to analyze a set of weights (in text form), since the attention to each weight in the text via key/query can sort of emulate the connection structure of the network that the weights represent.

▲

giancarlostoro 3 hours ago | parent | prev | next [-]

> I can interpret weights too, but will that interpretation be correct?

Chances are much higher than a model can figure out what the patterns are, decipher them, and retain that knowledge in a meaningful way it can reason about. Me and you might take years or months to do all of that on our own, with a "map" of the weight we might be able to discuss it, but not as effortlessly as an LLM.

Honestly, I had told a few different colleagues that reading a model is probably not that exclusive to frontier models. It's probably yet another reason OpenAI / Anthropic don't release any big open weight models, but thankfully Google and Microsoft release reasonably sized models. Imagine a new GPT open weight release, it would get dissected drastically.

▲

mwernolagoo 3 hours ago | parent | prev | next [-]

Further than that, aren't "weights" just another sequence of tokens? We see them as vectors, pass to the API or as strings into a chatbox interface. Is it so surprising that the high-dimensionality linear algebra machine crunches them and gives us back useful vectors that can be mapped to text?

Isn't that what this whole technology paradigm is about? How is it surprising?

Granted, it is cool. But I see so much of "look at this cool thing I did with LLMs" and honestly... we need more structured text, more explicit documentation, more ergonomic and intuitive CRUD applications.

The amount of "look at cool things" is growing way faster than before and they are all sort of low effort now because the "agentic" LLM-VM looping apparatus can do a lot of heavy lifting and run with our prompts.

And I mean, if one has jargon, knowledge, and intention, they can also leverage that into great structured text, ergonomic and intuitive CRUD apps etc, all of which add great value to our mundane pursuits of applied research, software products and services and so on.

Honestly, I begin to think the problem is people dazzling themselves instead of doing the housekeeping that is the bulk of life. Technology choice is entirely secondary.

▲

D-Machine 2 hours ago | parent [-]

> Further than that, aren't "weights" just another sequence of tokens?

No, not at all. I recommend you learn some of the basic linear algebra about how these things work before engaging in wild and completely nonsensical speculation.

▲

mlyle 2 hours ago | parent [-]

I understand linear algebra. It seems like he probably does, too.

Are you sure you have read and understood his point?

It seem quite possible that the flattened-model-weights ingested into a model have a much greater fidelity of representation in a similar model's KV cache than they do in a human's working memory, and that being able to loop and digest and chunk makes it unsurprising that models can excel at this type of task.

I think that's the point the person is making. Do you disagree? If so, why?

▲

D-Machine 2 hours ago | parent [-]

Weights are not sequences of tokens in any sense. This is pure nonsense. We don't "see them" [presumably, the weights] as vectors (they are matrices or tensors).

There is nothing obvious or expected about weights being easily explainable or being enough to explain, because the weights are static: you need to look at the inputs, the activations, and the weights all together, you can never understand a model with the weights alone, unless it is a trivial model. The person isn't making any coherent points at all.

There are also deep reasons to expect hard limits in intrepretability in general: https://www.seangoedecke.com/alignment-is-impossible/#why-me....

EDIT:

> It seem quite possible that the flattened-model-weights ingested into a model have a much greater fidelity of representation in a similar model's KV cache than they do in a human's working memory,

Bringing up the KV cache is a sign you don't understand what you are talking about here. The KV cache is a computational efficiency thing and has no impact on how these models actually work (and isn't needed at all). The vague analogy to human working memory also just takes the cake here.

> and that being able to loop and digest and chunk makes it unsurprising that models can excel at this type of task.

This is how it does... all tasks.

	▲	mlyle 2 minutes ago \| parent \| next [-]
		> Bringing up the KV cache is a sign you don't understand what you are talking about here. The KV cache is a computational efficiency thing and has no impact on how these models actually work (and isn't needed at all). The vague analogy to human working memory also just takes the cake here. You seem to really like to tell people that they don't know what they're talking about more than you seem to be parsing what they're actually saying. I remember when I used to look for reasons to pounce on people for not knowing what they were talking about based on a superficial thing or two rather than try to understand what they were saying. I mostly got past it as a teenager, though.
	▲	tomrod 7 minutes ago \| parent \| prev [-]
		A vector is a type of a matrix is a type of a tensor.

▲

lukan 3 hours ago | parent | prev | next [-]

And this reminds me of the king in the little prince, who had the power to order the sun to rise or go down. But only once a day at a certain time.

▲

huijzer 2 hours ago | parent | prev | next [-]

Biblically speaking, spirits from the vasty deep must be demons since they are from below and angels, God and Jesus are from above. So in that sense it’s not very impressive. Demons are something a sane person would rather avoid.

	▲	StableAlkyne an hour ago \| parent \| next [-]
		I mean, if you want to inject religion into this, that interpretation that it "must be demons" isn't even supported by the source text. In 1 Samuel, King Saul visits a medium in Endor in order to speak to the (dead) prophet Samuel. Human dead go to Shoel (in familiar terms, it is similar to the Greek underworld) in the Old Testament. Later on in the New Testament, the idea of a really really bad place is introduced in the form of Tartarus, which is specifically where the rebelling angels were sent. But that is not the same place human dead go. The idea that everything "not Heaven" must be Hell is an invention, caused mostly by dogma that rose from bad translations of the Greek and Hebrew in the middle ages (for example, King James - itself mostly correcting minor errors of the older Bishop's Bible - refers to all of it as Hell even though the original text most inarguably refers to different places different beings go).
	▲	snapcaster 2 hours ago \| parent \| prev [-]
		Crazy how much the quote went over your head. Literature written for peasants 400 years ago and you can't grasp the point

▲

coldtea 2 hours ago | parent | prev | next [-]

>I can interpret weights too, but will that interpretation be correct?

In TFA case, given that they were able to guess what the weights did (which the researchers knew in advance and could cross-check) and to even tweak and improve them, the answer is yes.

▲

keeganpoppen an hour ago | parent | prev | next [-]

interpretation is, as they say, in the eye of the beholder…

▲

ChrisClark 2 hours ago | parent | prev [-]

But the whole point of the article was that it was correct.

Why do this without testing? Why would you assume that?