13 minutes in Andrej is talking about how the models don't even really need the knowledge, it would be better to have just a core that has the algorithms it's learned, a "cognitive core." That sounds awesome, and would shrink the size of the models for sure. You don't need the entire knowledge of the internet compressed down and stashed in vram somewhere. Lots of implications.