How would we do that? There's no historic precedent for that. Its fundamentally an information theory thing: what's the max amount of intelligence you can get out of 1 KB/MB/GB? There has to be a limit and I'm not convinced that it's far off.

Once I start a conversation about Postgres query plans, maybe 95% of the knowledge will be almost certainly not needed, and for an inference provider there will be many more concurrent queries with reasonably large overlap. Maybe future architectures will be able to take advantage of this so not all parameters are needed in memory for almost all instances.

If we are talking about all knowledge, then I agree that the compression ratio is very impressive already.

All the terms are squishy, but being sloppy about it, I think there's intelligence that needs to be in the model, and knowledge that could live in a database. Right now, models are memorizing a lot of stuff they don't need to. We know how to index lots of information on (comparatively slow) SSDs or across a network.

See Karpathy's "Cognitive Core" idea. (I don't have a good link)

So far, frontier capability keeps getting shrunk to fit consumer level hardware. The lag time being over a year, though, precludes this from being called SOTA by the time it arrives.

There must be a limit, I agree, but there have been no signs of approaching it yet. The most recent cohort of small models have shown the biggest leap in capability so far.

It doesn't matter. However much intelligence you can squeeze into 1 GB, people will always want more.

Standard Def TV was plenty for 50 years. But when more was on offer, everyone went for it, and now you can't even buy a 480p TV.

Historical precedent: Quickselect, or any other discovery of a surprising algorithmic speed-up.

LLMs haven’t been on the scene for very long. There’s still lots of room for efficiency discoveries. Plus, I keep hearing quantum is going to be a big deal in the next few years.

I haven't tried it, but colibri is meant to allow running GLM 5.2 the 744B model in 32GB by using prefetches and streaming from fast SSDs.

https://github.com/JustVugg/colibri

I'm claiming that the max amount of intelligence you get out of a 100GB model is unlikely to be that much lower than what you can get out of a 1TB model.

It’s not just about small models, that’s only one part of evolution

Some groups are baking models into silicone, Deepmind has an example, it gets 18,000 tokens/sec on Llama 3.1, not sure about parameter size

> How would we do that? There's no historic precedent for that. Its fundamentally an information theory thing

Maybe? Human science history is absolutely littered with examples of things that were “constrained” by a fundamental law … until they weren’t, because we’d misunderstood or misapplied the law.

> How would we do that? There's no historic precedent for that. Its fundamentally an information theory thing: what's the max amount of intelligence you can get out of 1 KB/MB/GB? There has to be a limit and I'm not convinced that it's far off.

Sure, it might be very near using the current approach, but... it might also be might be very far off because we are using the wrong approach.

I mean, look at the max amount of intelligence you can get out of a human brain powered by two bananas...

Just replicate dogs brain. It is the ultimate state of art (far smarter than people). But you need a few kb of memory to replicate resoning skills.

Dogs are very smart!