I think this can't work because an LLM needs too much data, and before the internet there probably just wasn't enough to get close to what we have now

▲

jvanderbot 2 hours ago | parent | next [-]

Even simpler: Can GPT-2 anticipate and build Gwen/Deepseek? I think the answer is almost trivially "no", so I wonder what changed?

▲

ben_w an hour ago | parent [-]

Lots of things changed, GPT-2 is small (1.5e9) and is also a base model, so it is only doing next-token/autocomplete rather than prompt-response like even the first ChatGPT-3.5 was doing.

▲

jebarker an hour ago | parent [-]

Just for the sake of clarity: all LLMs up to today are still only doing next-token/autocomplete. The training process got additional stages to shape the model weights, but standalone LLMs are still deployed essentially identically.

▲

ben_w 37 minutes ago | parent [-]

If you gave GPT-2 a question and ended with a "?", it might answer, but also it might write several more questions in a similar category.

IMO, the mechanism isn't the important thing, the behaviour is. If you look at the step-by-step, we are also looking for the next word or motor action (and for whoever is about to suggest that we humans plan ahead, Transformer-based LLMs have been shown to also do this); as this is not a useful description of what it means to be a living brain, I'd say it's also not a useful description of what makes everything post-InstructGPT different from what came before.

	▲	jebarker 34 minutes ago \| parent [-]
		I agree completely - behaviorally the models have changed drastically due to RLHF, RLVR and now maybe even more so due to agentic harnesses. But the mechanism of prediction hasn’t changed, that was all I was clarifying.

▲

inigyou an hour ago | parent | prev | next [-]

Why couldn't an LLM, if it was smart enough, generate and consume its own data?

I know the answer: because it leads to model collapse. But why is that? Wouldn't a smart model not collapse? It's seeming like they keep getting smarter because we keep pouring more of our own knowledge into them, not because they are actually getting smarter. And yes, sometimes a dumb but persistent bruteforcer can make new discoveries.

	▲	Garlef an hour ago \| parent \| next [-]
		> if it was smart enough and i think this is exactly the crux; the really big models need really big datasets and current gen LLMs get a lot of training data beyond "all books + all of the internet" the objection is then that producing this additional data would already confound it with pre "virtual cutoff date" knowledge (since the training data probably implies mathematical and SWE concepts that were developed post "virtual cutoff date")
	▲	smusamashah an hour ago \| parent \| prev [-]
		If it is smart enough to generate data it can consume to train itself better, it is already smart enough to not need to do that.

▲

hackernudes an hour ago | parent | prev [-]

Maybe we can synthesize large amounts of limited information. I thought that new training data is mostly synthetic anyway.