Why isnt LLM training itself open sourced? With all the compute in the world, something like Folding@home here would be killer

▲

DesaiAshu 3 days ago | parent | next [-]

data bandwidth limits distributed training under current architectures. really interesting implications if we can make progress on that

▲

dogcomplex 2 days ago | parent | next [-]

Limits but doesn't prohibit. See https://www.primeintellect.ai/blog/intellect-3 - still useful and can scale enormously. Takes a particular shape and relies heavily on RL, but still big.

▲

andoando 2 days ago | parent | prev [-]

What bandwith limits? Im assuming the forward and backward passes have to be done sequentially?

	▲	DesaiAshu 16 hours ago \| parent [-]
		Yes also passing data within each layer

▲

mike_hearn 2 days ago | parent | prev | next [-]

It is in some cases. NVIDIA's models are open source, in the truest sense that you can download the training set and training scripts and make your own.

▲

throwaway27448 2 days ago | parent | prev | next [-]

It's either illegal or extremely expensive to source quality training material.

	▲	m4rtink 2 days ago \| parent [-]
		Yeah, turns out if you want to train a model without scrapping and overloading the whole of Internet while ignoring all the licenses and basic decency is actually hard & expensive!

▲

doctorwho42 2 days ago | parent | prev [-]

Well it is, it's in the name "OpenAI". /S