Yes, the Chinchilla scaling laws from 2022: https://arxiv.org/abs/2203.15556 More parameters directly and mechanically means less prediction error for each token.

(This moves the argument to "is being really good at predicting the next token the same as being intelligent?" which has happily diverted many minds for years now)

▲

luke5441 2 days ago | parent | next [-]

From your link:

> for every doubling of model size the number of training tokens should also be doubled

Well, I don't think many claim that increasing the model size while also increasing the training tokens doesn't make sense.

The problem is how to get the training tokens...

	▲	stymaar 2 days ago \| parent [-]
		Interestingly enough labs do train with 100s or even 1000s times more tokens than what's proposed by the paper as an optimal value. And the answer to your question is basically just “synthetic data”.

▲

stymaar 2 days ago | parent | prev [-]

The Chinchilla paper is probably the poster child of “paper that aged like milk” in LLM field though. Everyone over the past 4 years have trained with several orders of magnitude more data than the Chinchilla optimal number.

	▲	sroussey 2 days ago \| parent [-]
		If you don’t know which data has the highest value, though more at it. Chinchilla is probably still true, but with that caveat. And it could be like advertising… you know a lot of it is worthless, but you don’t know which part.