I'm over simplifying here but graph database and knowledge graphs exist. An LLM doesn't need to preserve everything in context, just what it needs for that conversation.

Unless there is a trick that I am missing, I don't think this will work by itself. The fundamental thing is what can the model attend to as it generates the next token.

If you give a summary+graph to the model, it can still only attend to the summary for token 1. If it's going to call a tool for a deeper memory, it still only gets the summary when it makes the decision on what to call.

You get the same problem when asking the model to make changes in even medium-sized code bases. It starts from scratch each time, takes forever to read a bunch of files, and sometimes it reads the right stuff, other times it doesn't.