This is fun, but historically people have gone a bit overboard with saying that models like this, or n-gram language models, are anywhere close to large neural network models. There is certainly a connection though.

▲

Matumio 4 hours ago | parent [-]

Yes, but it is a useful insight that both methods try to solve the same mathematical problem. It's better than thinking of LLMs as magic.

When you say "cross-entropy loss" people without stats background go to Wikipedia, take a glance, and adjust their mental model to "inscrutable magic".

Thinking of the main difference as the trade-off in how much CPU, memory and storage is allowed is not really wrong.

The part that is wrong is to think of gzip as a method that might reach similar complexity or generalization. And more importantly, to ignore the advanced way how training data gets curated or generated for (instructed, chain-of-thought) LLMs. But even then. The mental model that the LLM's goal is text compression is not wrong. The question to ask next is what kind of text it is expecting to compress.

	▲	montebicyclelo 2 hours ago \| parent [-]
		Yep agree, good details, and this doesn't contradict my point above, about people going "overboard" with the comparison. I do think when making these comparisons, it is worth emphasising that neural nets are really different. E.g. I used to see people equating LLMs to n-gram models, etc. which is overly simplistic, (especially in the early days when the models weren't as good).