There’s also the difference between a model recursively improving “itself” and improving itself via online learning.

The former being that these models are helping develop and train future models, but they might not veer too far off in architecture (yet).

The latter is a model being able to train/learn on the fly, in real time, permanently (not just in the current conversation/session), or in other words, adjusting/managing its own weights. But, it also seems like it would take an entire paradigm shift in model architecture from what most LLMs are built on, but I could be wrong.

You may be interested in TITANS:

Test-Time Learning: The model updates its own memory weights while running an inference task.