I suspect that context can’t fully replace a mental model, because context is in-band, in the same band, as all input the LLM receives. It’s all just a linear token sequence that is taken in uniformly. There’s too little structure, and everything is equally subject to being discarded or distorted within the model. Even if parts of that token sequence remains unchanged (a “stable” context) when iterating over input, the input it is surrounded with can have arbitrary downstream effects within the model, making it more unreliable and unstable than mental models are.

Okay I see now. I'm just shooting in the dark here, if there's an ability to generate the next best token based on the trained set of words. Can it be taken a level up, in a meta level to generate a generation? like genetic programming does. Or is that what the chain of thought reasoning models do?

Maybe I need to do more homework on LLMs in general.