This is fascinating, and makes me wonder what other things that 'should' be impossible might just be waiting for the right configuration to be tried.

For example, we take for granted the context model of LLMs is necessary, that all you can do is append and anything that changes the beginning requires a recalculation of whatever comes after it. And that does match how training works.

But all sorts of things would become possible if it were possible to shift things in and out of context without recomputing it all; conservatively you could avoid compaction, optimistically it might be a way to get info to the model that's both more deeply integrated than search and more efficient than training larger and larger models.