So, when you said “proven” I thought you might have meant like, a mathematical proof. It appears that this is not what you meant. (Right?) (If it is what you meant, I was asking for like, the particular theorem.)

Also, ok, when I said “intelligence”, it was because you were already talking about how “smart” the model could be. So, I thought you were already on board with using the word “intelligence” to refer to the phenomenon where these kinds of models produce outputs that satisfy the kinds of tasks they are pointed at.

None of those links give an argument that the current 1GB models are the best they can be.

My understanding is that so far when training a model by distillation (using the logits of the teacher model), one can achieve better outcomes than one could if training the 1GB model from scratch on the same training data as the large model, and that so far, better models as the teacher model have yielded better results for the student model.

▲

himata4113 11 hours ago | parent [-]

It is infact mathematical proof that is somewhat well understood although specifics are debated. We do in fact know that the limit follows information density law which is something we can measure and we are pretty close to it.

▲

drdeca 8 hours ago | parent [-]

Well, what’s the theorem statement for the result you’re thinking of then?

Of course n bits can only encode for 2^n options. But to prove mathematically that we are close to the best that can be achieved in 1GB requires a mathematical definition of what we mean by better. Rather, it requires at least a choice of a proxy for what we mean by better. (Showing that some approximation of what we mean by better is close to as good as can be, would suffice. Like, if we can show that the loss can’t get much lower with a 1GB model, that would count.)

	▲	himata4113 5 hours ago \| parent [-]
		We generally don't care enough to test it since it would be a significate waste of resources that's why we have models in the trillions of parameters instead of continuing to train smaller models smaller models hit a wall and generally stop improving and start overfitting and generalization starts to degrade it's less of a theorem, but an observation that has occured during training and documented here and there across many papers.