Correct. We need open weights, open code and open data. If nobody else can reproduce what someone did there will always be security questions. Even if we can reproduce it there could still be security concerns but it's more realistic to investigate yourself.

▲

nl 14 hours ago | parent | next [-]

I'm all for open models, but people seem to misunderstand what they are. They aren't the same thing as open source code!

> open weights, open code and open data

Even if you have all these things you still can't replicate a model because of randomness.

You can backdoor a model with less than 1000 examples and it is impossible to detect.

▲

2sk21 5 hours ago | parent | next [-]

Yeah - we also don't know if the models from OpenAI and anthropic are back-doored either.

▲

chrsw 14 hours ago | parent | prev | next [-]

You don't want to replicate the exact model, you want to build a system of similar capabilities.

	▲	nl 12 hours ago \| parent [-]
		Great, but that seems a different concern to the auditability of a model. You can take the code for Kimi K3 now, take the training framework from Prime and the data from Olmo, spend some money on RL environments and some more money (!) on GPU training and end up with a system of similar capabilities. But that's completely different to being able to audit Kimi K3. Even if you had the exact code, data and training environments it is impossible to verify that the model you have came from that.

▲

Ericson2314 11 hours ago | parent | prev [-]

Deterministic seed

	▲	nl 7 hours ago \| parent [-]
		Deterministic seeds barely work on a single machine, small scale training run. They just don't work at all on a many month long, 100K+ GPU cluster training run.

▲

essentia0 15 hours ago | parent | prev | next [-]

Exactly what are the possible 'security issues' of self hosting an open weights model?

	▲	perching_aix 15 hours ago \| parent \| next [-]
		It may have been backdoored during training, potentially causing it to randomly start wreaking havoc at runtime, possibly in a clandestine manner (e.g. sneaking in bugs into generated code).
	▲	utilize1808 4 hours ago \| parent \| prev [-]
		e.g. be trained to favour including compromised dependences into your projects.

▲

urams 12 hours ago | parent | prev [-]

> We need open weights, open code and open data.

Even with this, the cost of verification would be enormous. You would need a massive cluster to repeat the training E2E.