new | show | ask | jobs Github

bluegatty 15 hours ago

Making an LLM from raw data is value-add.

Distillation is just value extract.

It's soft, and I'm not sure what the answer should be ... but I think that there is a difference.

I think we start by recognizing that ... and then try to figure it out from there.

'The Internet' may be a public good, maybe we make them pay a tax for that, but that's different than distillation.

▲

nemomarx 14 hours ago | parent | next [-]

What makes the Internet raw data in a different way? wasn't it mostly worked on by people first?

▲

bluegatty 14 hours ago | parent [-]

There is value add in AI irrespective of how the data got to what it is.

Literally the biggest thing of our generation - AI - is the living embodiment of that 'value add' writ large.

'What is the difference' - is the AI you use all day, in comparison to 'all the world's data' you can use for stuff and do 'whatever' with it, but are not likely to come up with something hugely useful otherwise. Maybe, not likely, if you did, it would be 'value add'.

▲

nemomarx 14 hours ago | parent [-]

Okay, so if the chinese models are used everyday, do they become a value add? Like what's the line you're drawing here. Amount of value it creates?

▲

bluegatty 12 hours ago | parent [-]

Designing and creating an LLM from nothing is a monumental feat of Engineering and 'value add'.

Copying something is not.

Programming Microsoft Word is value add, copying the code is not.

Copying design ... there are some question marks there.

It's extremely easy to understand at it's core.

What makes it hard, is that faux intellectuals like to deconstruct ideas at the margins, and have those critiques stand in for reason.

"At sunrise the sun is only 'half there' ... there fore there is no 'day and night' just a blur! Day and night are the same thing!"

The training data used is part of all of this is a separate but related question.

	▲	inigyou 3 hours ago \| parent [-]
		they all just copied the Transformers paper anyway

▲

scotty79 15 hours ago | parent | prev [-]

> Making an LLM from raw data is value-add. > Distillation is just value extract.

There is a value-add in selecting the valuable parts out of the garbage. And let's face it. Largest models contain a lot of garbage.

	▲	bluegatty 14 hours ago \| parent [-]
		I think that's kind of fair, but it still fits within the context of 'some things are value add' and 'more or less than others'. We ought to identify that and integrate that into our thinking.