I think the end state of any productive LLM includes both: - a real-time knowledge/context graph that keeps track of state - continuously updating weights

Long term I think it's probably more interesting to explore continuously updating weights. But (and this may be yet another opinion that falls victim to the bitter lesson) even if it's _theoretically_ possible to encode everything in weights alone, I'm not sure it's not the most optimal. Today it's too expensive & finnicky to get better on your own tasks without overfitting (ie [0])

Practically (like next few months) getting the right instructions & context would be a huge improvement in capabilities.

[0] https://thinkingmachines.ai/blog/on-policy-distillation/#tra...