We already have a 1gb model that is as capable as it will ever be, there's a proven ceiling that cannot be passed. For example: you can't make a mice-sized brain as smart as a human brain no matter how hard you try.

▲

Philpax a day ago | parent | next [-]

Please source this claim. What 1GB models are capable of has increased generation-on-generation.

> For example: you can't make a mice-sized brain as smart as a human brain no matter how hard you try.

Sure. We don't know where the ceiling is for our digital minds, though.

▲

himata4113 a day ago | parent [-]

They have not increased in capabilities, they have increased in specialization.

If you train a small model in another domain it will begin losing capabilities in the former domain. This is effectively the sigmoid problem.

Although I will admit that if we discover a higher information density algorithm that it might change, but not by a substantial amount to where "super intelligence" in 1gb would be possible.

▲

Philpax a day ago | parent [-]

Over the last two years, this weight class has doubled its scores and/or saturated several benchmarks in the Qwen lineup alone without loss of generality: https://claude.ai/public/artifacts/9f249169-3623-417e-86cd-7...

There is undoubtedly a limit somewhere (there is only so much you can pack into a given size) but it's really not particularly clear where that limit is. I don't think it's superintelligence - that much I agree with you - but I think "We already have a 1gb model that is as capable as it will ever be" is strictly false.

▲

himata4113 a day ago | parent [-]

The measured entropy of the model remains nearly unchanged though which means we have lost capabilities we have not measured, the model hasn't become "denser" it just became more specialized.

It's like comparing two person A and B of similar intelligence where A is smarter and B is a genius at signing, but signing was not on the test so person A won.

▲

Philpax a day ago | parent | next [-]

Assumes facts not in evidence. Please show your working.

▲

himata4113 a day ago | parent [-]

https://en.wikipedia.org/wiki/Model_collapse - you want to use sigmoid 1.0, but the closer you are to 1.0 the higher the chance your model will collapse so you use 0.99-0.98, but those lead to data loss so after n passes all the original data becomes lost so you have a strict data limit there.

The rest is just the general reality I am sure you are familiar with:

- https://en.wikipedia.org/wiki/Catastrophic_interference

- https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)

- https://en.wikipedia.org/wiki/Entropy_(information_theory)

	▲	Philpax a day ago \| parent [-]
		You... haven't shown any evidence that we're near collapse. That's what I'm asking you for. Show me some evidence that we are losing capabilities with we have today.

▲

neuroticnews25 17 hours ago | parent | prev [-]

> The measured entropy of the model remains nearly unchanged though which means we have lost capabilities

Or we've lost random noise, or redundancy not captured by the entropy measurement.

▲

drdeca a day ago | parent | prev | next [-]

What proof of a ceiling are you talking about? Wouldn’t proving this require a good definition for intelligence, which I don’t think there is consensus on?

▲

himata4113 a day ago | parent [-]

- https://en.wikipedia.org/wiki/Model_collapse

- https://en.wikipedia.org/wiki/Catastrophic_interference

- https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)

- https://en.wikipedia.org/wiki/Entropy_(information_theory)

As for intelligence, the only way we have that is by allowing the model to fill the blanks which have to come from the training data. The models cannot have true intelligence for as long as they are linear models, what we see with reasoning is "boxed" intelligence where the models are effectively "modifying" themselves by feeding it's own reasoning data back into input deriving most plasible output given known information. However, the model is not able to retain what it has learned therefore that intelligence is gone the moment the session is 'full'. You can go pretty far by continiously distilling discovered information, but again all that has to come from the original training data and models own outputs, which it has to take for granted as the 'intelligence' gained is lost creating what we see is the maximum possible benchmark performance and why smaller models are not able to score as high while theoretically having the same capabilities. We can see this with larger models where they can solve tasks much faster than smaller ones as it does not require to generate the solution due to the fact that the solution is already in the training data as 'baked' intelligence and it doesn't have to 'create' it during reasoning.

▲

drdeca a day ago | parent [-]

So, when you said “proven” I thought you might have meant like, a mathematical proof. It appears that this is not what you meant. (Right?) (If it is what you meant, I was asking for like, the particular theorem.)

Also, ok, when I said “intelligence”, it was because you were already talking about how “smart” the model could be. So, I thought you were already on board with using the word “intelligence” to refer to the phenomenon where these kinds of models produce outputs that satisfy the kinds of tasks they are pointed at.

None of those links give an argument that the current 1GB models are the best they can be.

My understanding is that so far when training a model by distillation (using the logits of the teacher model), one can achieve better outcomes than one could if training the 1GB model from scratch on the same training data as the large model, and that so far, better models as the teacher model have yielded better results for the student model.

▲

himata4113 10 hours ago | parent [-]

It is infact mathematical proof that is somewhat well understood although specifics are debated. We do in fact know that the limit follows information density law which is something we can measure and we are pretty close to it.

▲

drdeca 7 hours ago | parent [-]

Well, what’s the theorem statement for the result you’re thinking of then?

Of course n bits can only encode for 2^n options. But to prove mathematically that we are close to the best that can be achieved in 1GB requires a mathematical definition of what we mean by better. Rather, it requires at least a choice of a proxy for what we mean by better. (Showing that some approximation of what we mean by better is close to as good as can be, would suffice. Like, if we can show that the loss can’t get much lower with a 1GB model, that would count.)

	▲	himata4113 4 hours ago \| parent [-]
		We generally don't care enough to test it since it would be a significate waste of resources that's why we have models in the trillions of parameters instead of continuing to train smaller models smaller models hit a wall and generally stop improving and start overfitting and generalization starts to degrade it's less of a theorem, but an observation that has occured during training and documented here and there across many papers.

▲

mathieudombrock a day ago | parent | prev [-]

What model is that?