> There are grammar rules

And they're made out of weights.

As opposed to integers in normal programming.

The 'magic' in weights is that the rules are spread through the whole model and you can't point to one place which encodes them.

The grokking paper shows that this stops being the case with enough training data and enough compute.

▲

throw310822 11 days ago | parent [-]

Integers in normal programming represent data or instructions; instructions are hand coded, have rigidly defined semantics, are not differentiable and have no redundancy.

> The 'magic' in weights is that the rules are spread through the whole model ... The grokking paper shows that this stops being the case with enough training data and enough compute.

I don't understand what you mean to say. That weights are not magic? That weights are not weights? NNs are made up of weights, which are learned and not coded. The fact that they do learn world models (grammar rules in your example), and that these models' weights tend to roughly concentrate by function and level of representation is perfectly logic but even more amazing. (Notice that much of the dismissive attitude towards LLMs depicts them as pure syntactic manipulators without the ability to develop world models- the exact opposite of what you point out).

▲

noosphr 11 days ago | parent [-]

>Integers in normal programming represent data or instructions; instructions are hand coded, have rigidly defined semantics, are not differentiable and have no redundancy.

I can, and have, written programs using an evolutionary algorithm that then run on bare metal. None of the things you list are true for those programs, yet other than being computationally more expensive to train they work just as well as neural networks.

>I don't understand what you mean to say

The diffusness of weights across the whole model isn't an innate feature of deep learning models. It is a feature of sparse training data and little compute.

	▲	throw310822 11 days ago \| parent [-]
		You're nitpicking against the line: "The weights make the words. Are you understanding me? We opened it up. There's no dictionary in there, no grammar rules, no little man. Just weights. Eighty layers of numbers getting multiplied together." In this context "there's no grammar rules" means "no separately hand-coded grammar rules". Everything is made up of weights, and the fact that weights that end up encoding for grammar rules tend to concentrate in particular locations (without being self-contained- there is no hard boundary) rather than uniformly diffused through the model is irrelevant to the matter. It seems you're arguing against a diffuseness requirement that is not in the text.