Further than that, aren't "weights" just another sequence of tokens? We see them as vectors, pass to the API or as strings into a chatbox interface. Is it so surprising that the high-dimensionality linear algebra machine crunches them and gives us back useful vectors that can be mapped to text?

Isn't that what this whole technology paradigm is about? How is it surprising?

Granted, it is cool. But I see so much of "look at this cool thing I did with LLMs" and honestly... we need more structured text, more explicit documentation, more ergonomic and intuitive CRUD applications.

The amount of "look at cool things" is growing way faster than before and they are all sort of low effort now because the "agentic" LLM-VM looping apparatus can do a lot of heavy lifting and run with our prompts.

And I mean, if one has jargon, knowledge, and intention, they can also leverage that into great structured text, ergonomic and intuitive CRUD apps etc, all of which add great value to our mundane pursuits of applied research, software products and services and so on.

Honestly, I begin to think the problem is people dazzling themselves instead of doing the housekeeping that is the bulk of life. Technology choice is entirely secondary.

▲

D-Machine 2 hours ago | parent [-]

> Further than that, aren't "weights" just another sequence of tokens?

No, not at all. I recommend you learn some of the basic linear algebra about how these things work before engaging in wild and completely nonsensical speculation.

▲

mlyle 2 hours ago | parent [-]

I understand linear algebra. It seems like he probably does, too.

Are you sure you have read and understood his point?

It seem quite possible that the flattened-model-weights ingested into a model have a much greater fidelity of representation in a similar model's KV cache than they do in a human's working memory, and that being able to loop and digest and chunk makes it unsurprising that models can excel at this type of task.

I think that's the point the person is making. Do you disagree? If so, why?

▲

D-Machine 2 hours ago | parent [-]

Weights are not sequences of tokens in any sense. This is pure nonsense. We don't "see them" [presumably, the weights] as vectors (they are matrices or tensors).

There is nothing obvious or expected about weights being easily explainable or being enough to explain, because the weights are static: you need to look at the inputs, the activations, and the weights all together, you can never understand a model with the weights alone, unless it is a trivial model. The person isn't making any coherent points at all.

There are also deep reasons to expect hard limits in intrepretability in general: https://www.seangoedecke.com/alignment-is-impossible/#why-me....

EDIT:

> It seem quite possible that the flattened-model-weights ingested into a model have a much greater fidelity of representation in a similar model's KV cache than they do in a human's working memory,

Bringing up the KV cache is a sign you don't understand what you are talking about here. The KV cache is a computational efficiency thing and has no impact on how these models actually work (and isn't needed at all). The vague analogy to human working memory also just takes the cake here.

> and that being able to loop and digest and chunk makes it unsurprising that models can excel at this type of task.

This is how it does... all tasks.

	▲	mlyle a minute ago \| parent \| next [-]
		> Bringing up the KV cache is a sign you don't understand what you are talking about here. The KV cache is a computational efficiency thing and has no impact on how these models actually work (and isn't needed at all). The vague analogy to human working memory also just takes the cake here. You seem to really like to tell people that they don't know what they're talking about.
	▲	tomrod 6 minutes ago \| parent \| prev [-]
		A vector is a type of a matrix is a type of a tensor.