As an industry, I wish we would stop calling these things "open weight" because it is too easy to confuse with actual "open source", which they are not.

Photoshop source code+ OSI license = open source

Photoshop binary you can run on your own computer = open weight

Photoshop SaaS web app = closed, proprietary (Opus, GPT, etc.)

"Open weight" models are still just binary blobs that are completely inscrutable. It's like bringing home a dog from the rescue and just hoping that it doesn't have a tendency to bite kids in the face. You just can't know. The only thing that you can do is try to add more training (fine tuning) telling it not to bite kids.

I don't think the FOSS community has ever accepted this, but somehow we're feeling like it is okay now.

Photoshop source code+ OSI license = open source

Photoshop binary you can run on your own computer = open weight

I don't think this is a correct analogy. You are not allowed to distribute modified versions of the Photoshop binary. Most open weight model licenses allow you to make and distribute your own finetunes, etc.

Given an open weights model trained to sometimes bite kids, we can’t train it to not bite kids, even though billions of dollars of research have been thrown at this open problem.

Given an open weights model trained to never bite kids, you can get it to bite kids with 10 prompts and a linear projection, the known simple algorithm doesn’t even need a backwards pass.

yay asymmetry!

I believe that comparing LLMs with traditional deterministic software is fundamentally misleading. It is extremely difficult to truly interpret what LLMs do internally, and as of now, nobody fully understands it. Even if you trained the LLM yourself, there is no source code you can simply read and learn from.

Sure, having information about how these models were trained is helpful for reproducibility, but it is basically impossible for anyone without substantial capital and access to the same (likely copyrighted) data to reproduce the model. For normal users, owning the model weights essentially means owning 100% of the model, you can inspect and study the weights in much the same way as the lab that produced the model can, you can modify the weights, and you can use and distribute them if the license allows you to

It is useful to indicate you can run the weights on your own hardware. That’s categorically different from most other commercial offerings. It’s as if your adobe example ignores the reality that would exist had photoshop been invented in 2019: cloud only.

This analogy is terrible and seems to be extremely misinformed about how rescues evaluate dogs before they are put up for adoption