The models are not open source. They are deeply proprietary since we have no access to the source materials and cannot reproduce the model independently. They are opaque binary blobs that the Chinese labs are just allowing other providers to run directly instead of only access through an API.

▲

0cf8612b2e1e 4 hours ago | parent | next [-]

Do you expect any of the labs to have an accompanying data dump with: here’s every book ever written, newspaper article, song lyric, Disney movie, GitHub repo, etc. Oh, and we obviously never paid for any of this.

Even if you did, I doubt training is bit-for-bit reproducible, so you will always have to take someone’s word for the final artifact.

	▲	anon373839 16 minutes ago \| parent [-]
		These OSS purity spirals are not helpful. Fully open source LLMs are great resources, but they underperform for the reason you stated: they can't use the same training data. Surely the people making these complaints know this. So I find myself wondering: are they just really pedantic? Or is this issue getting astroturfed?

▲

rstuart4133 4 hours ago | parent | prev | next [-]

> we have no access to the source materials and cannot reproduce the model independently.

Given the USA companies have been loudly claiming the Chinese models are distillations of their models, also claiming "no access to source materials" seems dubious. As it was dubious anyway with because the Chinese publish lots of papers on how their models are designed, I'm left feeling I'm looking at the south end of a north bound bull.

	▲	usef- 3 hours ago \| parent [-]
		Distillation of this kind isn't the only data that is fed into models, it's still a small minority. It might be weighed higher in training to learn the thinking patterns/etc but you need a lot more tokens.

▲

matheusmoreira 5 hours ago | parent | prev | next [-]

Which is why we need the ability to train our own models. Maybe it will be viable to do it in a distributed computing setup one day. Research's already being done in that direction.

▲

verdverm 5 hours ago | parent | prev [-]

They are more than opaque blobs, some examples:

- One can load them up in a model explorer to see the layers and other components, how it is designed

- One can fine tune the models, which requires adding LoRA to the model and then running some training iterations

▲

nwiswell 4 hours ago | parent [-]

You can make parallel arguments for binary executables.

▲

verdverm 4 hours ago | parent [-]

model weights are more like a video file which has metadata about the format so any video player can play it

we run and change llm models with a variety of tools

▲

nwiswell 4 hours ago | parent [-]

Here's my thinking:

Cloud:

- You cannot directly execute a remotely-hosted program.

- You cannot run inference on an API-served model.

---

Closed-source:

- You can execute a program with the binary. You cannot generate a new binary, but you could try to reverse-engineer it or (painfully) modify its execution.

- You can run inference on a model with the weights. You cannot re-produce a new set of weights from scratch, but you can fine-tune.

---

Truly open:

- You can freely modify the source and produce new binaries.

- You can use the original training data and model architecture to independently re-produce the weights (assuming you've got the compute). You can modify the model architecture to get the weights that would've resulted from training the model that way.

---

To me these are pretty clear parallels... I don't think the weights provided in a vacuum are in the spirit of open source, historically speaking.

The policy argument is totally separate, of course, and I fully understand why none of the frontier labs are truly open.

	▲	rstuart4133 4 hours ago \| parent [-]
		> Closed-source: You can execute a program with the binary. You cannot generate a new binary, but you could try to reverse-engineer it or (painfully) modify its execution. Time have changed. This should be: Closed-source: You point an LLM at it, and get back source that's often easier to understand than the original.