> Further than that, aren't "weights" just another sequence of tokens?

No, not at all. I recommend you learn some of the basic linear algebra about how these things work before engaging in wild and completely nonsensical speculation.

▲

mlyle 2 hours ago | parent [-]

I understand linear algebra. It seems like he probably does, too.

Are you sure you have read and understood his point?

It seem quite possible that the flattened-model-weights ingested into a model have a much greater fidelity of representation in a similar model's KV cache than they do in a human's working memory, and that being able to loop and digest and chunk makes it unsurprising that models can excel at this type of task.

I think that's the point the person is making. Do you disagree? If so, why?

▲

D-Machine 2 hours ago | parent [-]

Weights are not sequences of tokens in any sense. This is pure nonsense. We don't "see them" [presumably, the weights] as vectors (they are matrices or tensors).

There is nothing obvious or expected about weights being easily explainable or being enough to explain, because the weights are static: you need to look at the inputs, the activations, and the weights all together, you can never understand a model with the weights alone, unless it is a trivial model. The person isn't making any coherent points at all.

There are also deep reasons to expect hard limits in intrepretability in general: https://www.seangoedecke.com/alignment-is-impossible/#why-me....

EDIT:

> It seem quite possible that the flattened-model-weights ingested into a model have a much greater fidelity of representation in a similar model's KV cache than they do in a human's working memory,

Bringing up the KV cache is a sign you don't understand what you are talking about here. The KV cache is a computational efficiency thing and has no impact on how these models actually work (and isn't needed at all). The vague analogy to human working memory also just takes the cake here.

> and that being able to loop and digest and chunk makes it unsurprising that models can excel at this type of task.

This is how it does... all tasks.

	▲	mlyle a minute ago \| parent \| next [-]
		> Bringing up the KV cache is a sign you don't understand what you are talking about here. The KV cache is a computational efficiency thing and has no impact on how these models actually work (and isn't needed at all). The vague analogy to human working memory also just takes the cake here. You seem to really like to tell people that they don't know what they're talking about more than you seem to be parsing what they're actually saying. I remember when I used to look for reasons to pounce on people for not knowing what they were talking about based on a superficial thing or two rather than try to understand what they were saying. I mostly got past it as a teenager, though.
	▲	tomrod 6 minutes ago \| parent \| prev [-]
		A vector is a type of a matrix is a type of a tensor.