Yes -> every 18 months they've gotten 90% more efficient for the same level of quality for about 5 years. There's little sign that trend is slowing. If anything, there's reason to believe that System 1 models (plus potentially 1-2-3 workflows) may increase that over the next 3-5 years.

You'll know when the trend stops -> when the intelligence differential between smaller models like 7B starts to grow instead of shrink from 32B models -> that means 7B is getting about as smart as it can get. Then, 32B will follow next, then 70B, etc etc.

We haven't yet seen that at any size AFAIK.

▲

thefourthchime 6 hours ago | parent [-]

Andrej Karpathy said once that he expects superintelligence could fit in 1 billion parameters.

▲

onlyrealcuzzo 5 hours ago | parent [-]

Super intelligence that doesn't have to deal with the real world, maybe.

I wouldn't be surprised if less than 1B param equivalent of our brain deals with solving math and writing computer programs and physics and all the things we tend to associate with "intelligence" - especially if you ultra optimized for that, I doubt our brain works like that.

Dealing with the real world, I highly highly doubt it.

	▲	Gigachad 3 hours ago \| parent \| next [-]
		It would be interesting if running ends up being a more complex task than advanced math. And our brains are just 95% allocated to dealing with the real world.
	▲	jstummbillig 5 hours ago \| parent \| prev [-]
		How about if we get away from written text as the input, to something more fundamental, that then also is able to produce text (among other things)? Given that humans learn to talk while having encountered a measly number of word instances, and, given enough time, we should always be able to improve on the lottery that is biology, it does seems fairly likely.