Is this some form of rage bait? 2005 we hadn't the GPUs, we have today. There are other factors, but I think this is the big one. The mathematics of building an LLM are really old, we just hadn't the hardware to do the needed calculations.

▲

nonethewiser 4 hours ago | parent [-]

Right. Therefor it's not simply a derivative of information. The hardware is required to build the model. Software as well. The model uses information, it is not "distilled" from it.

"Distillation" literally means to separate and take some components out of something. You can distill how a model works from a model. You cant distill a model from information because the information does not contain the model.

People are happy to conflate distilling with building because they dont like how the information was used. You distill how the model works from the model, and you build a model with information. Both could be morally good or bad but its not the same thing.

▲

girvo 4 hours ago | parent | next [-]

Information is information. Why is some information considered different than others in your estimation?

▲

embedding-shape 4 hours ago | parent | next [-]

> Information is information

Not really, what the information actually is, matters a great deal. It's harder to get good results going from "nothing > model+weights" than "nothing + traces from known good sessions of other good model > model+weights", this is what the "distillation" part is referring to. If "information is information", you wouldn't even need to separate good from bad sessions while doing the training, which leads to somewhat obvious results if you don't.

▲

nonethewiser 4 hours ago | parent | prev [-]

Can you be more specific? I have no idea what you are trying to say.

To succinctly restate my point, you cannot distill a model from information because the model is not contained within that information. You can distill a model from another model.

	▲	margalabargala 3 hours ago \| parent [-]
		Their point is that "training" and "distillation" are essentially the same. The difference between the words is whether the source material is output from another model, vs being some original text.

▲

realusername 4 hours ago | parent | prev [-]

That argument is moot as distillation also requires a lot of hardware and software, if copying models was as easy as that, we would have hundreds of competing models.

	▲	nonethewiser 4 hours ago \| parent [-]
		No. Building models and distilling models both require the hardware and software. It doesn't mean building models is distillation.