new | show | ask | jobs Github

LtWorf a day ago

So windows is open source because the binaries are a lossy compression of the original source?

▲

mirekrusin 18 hours ago | parent | next [-]

You can't take windows binaries and continue development on them.

Model weight release is a snapshot/checkpoint you can take and resume training on new data, producing new model.

You don't need original training history to modify it further.

▲

petu 20 hours ago | parent | prev | next [-]

Before we worry about source code, Microsoft doesn't grant me rights to modify/redistribute/sell copy of Windows I have.

▲

NitpickLawyer a day ago | parent | prev [-]

Weights are not binary. A model is created at init time, with random values. After that, it is being modified using data. The key point is that the labs modify the models "as weights". That means that weights are the intended / preferred way of modifying a model. Which, coincidentally, matches the definition of source in Apache 2.0. There is no "higher level" place where editing takes place. It all happens in weight space. Through the license you get the same rights as the lab that created it: view, inspect, run, modify, re-release. That's it. That's the only thing a license can grant you.

The rest is semantics, misunderstandings, and FUD. A model released under an open source license is open source. Training data is lab knowhow / IP. Which, historically, has never been required for any open source release.

▲

frabcus a day ago | parent [-]

Well, you can't add or alter data in pre-training from just the weights. Which, as I understand it, means you can't fundamentally increase core knowledge or cognitive ability, only what the model likes to do with those. You can only post-train, and you're subject as a result to catastrophic forgetting.

To explain simply as far as I can tell (would love to be corrected) the large number of pre-training tokens only works because the documents are randomly ordered.

So if you e.g. took a foundation model with open weights, then tried post-training it all the new data since its cut-off period, it would then end up over-trained on that new data, and forget older things.

	▲	mirekrusin a day ago \| parent [-]
		As I live next to EPFL, I'll give you example from them: their Meditron-70B model is adapted to the medical domain from Llama-2-70B through continued pretraining. They took weights of Llama-2-70B and continued training on PubMed, medical guidelines and general data. Weights aren't just executable artifact that's consumed by users. Third parties actually use released parameter state as the editable starting point for further training and produce new foundation models from it.