If you think of LLMs as programs. The weights and inference code are very much a binary.

While the training code and data are the true source. Since if you want to robustly modify the LLM that's actually what you need.

But since "compilation" (training) is extremely compute intensive this isn't something accessible to anyone without an entire datacenter.

Anyway semantics aside having the binary is still infinitely better than dealing with an api as far as privacy and control go.

That’s not how it works though. Two training runs on the same data don’t produce the same weights. And if you want to modify the AI, you do so by fine tuning the weights not rerunning training. In every respect that matters, the weights are both the binary and the source code together.

The weights + the architecture are already 100% of the code, the transformer is just a mathematical expression + helper programs whose sources are provided. The transformer itself is not even a stateful program, so a it is no more a binary than Piet or Tromp's BLC are. It's merely incomprehensible. Training isn't compilation either, since training a model is closer to program induction and the data are samples defining the solution space.

I agree. But models in difference to compiled binaries, are useful as just weights and can be further refined and post-trained, at least.

I don't know LLM theory well enough to say if there's some secret sauce they can hold back that makes training ineffective. Less effective I'm sure, we don't have access to their smart training schemes, but post-training should always be possible IIUC.