But you don't have to hold the entire memory in context. You just need to perfect techniques to pull in parts of the context that you need. This can be done via RAG, multi-agent architectures, etc. It's not perfect but it will get better over time.

From my experience context window by itself tells half the story. You load a big document that’s 200k tokens and ask it a question, it will answer just fine. You start a conversation that soon enough balloons past 100k then it starts losing coherence pretty quickly. So I guess batch size plays a more significant role.

I'm over simplifying here but graph database and knowledge graphs exist. An LLM doesn't need to preserve everything in context, just what it needs for that conversation.

Context will need to go in layers. Like when you tell someone what you do for a living, your first version will be very broad. But when they ask the right questions, you can dive into details pretty quick.

Do you have some kind of tooling to automate the process? Would like to try it.

It's never quite that straightforward, or perceived as that straightforward. That's why most people just renew their insurance as it's easier than messing about changing and worrying if it will be any better. And how easy is it to transfer emails to another provider?

I have made 100MB of my own chat logs into a RAG memory and was surprised I didn't like using it much. Why? it floods the LLM with so much prior thinking that it loses the creative spark. I now realize the sweet spot is in the middle - don't recall everything, strategic disclosure to get the max out of AI. LLM memory should be like a sexy dress - not too long, not too short. You get the most creative outputs when you hide part of your prior thinking and let the mode infer it back.

I can easily see a world where users own the devices they buy and install the software they want, but the trend goes in the other direction.

in order for that lifetime companion, we'll need to make a leap in agentic memory.

Well, let’s take your life. Your life is about 3 billion seconds (100 year life). That’s just 3 billion next-tokens. The thing you do on second N is just, as a whole, a next token. If next-token prediction can be scaled up such that we redefine a token from a part of language to an entire discrete event or action, then it won’t be hard for the model to just know what you will think and do … next. Memory in that case is just the next possible recall of a specific memory, or next possible action, and so on. It doesn’t actually need all the memory information, it just needs to know that that you will seek a specific memory next.

Why would it need your entire database of memories if it already knows you will be looking for one exact memory next? The only thing that could explode the computational cost of this is if dynamic inputs fuck with your next token prediction. For example, you must now absolutely think about a Pink Elephant. But even that is constrained in our material world (still bounded physically, as the world can’t transfer that much information through your senses physically).

A human life up to this exact moment is just a series of tokens, believe it or not. We know it for a fact because we’re bounded by time. The thing you just thought was an entire world snapshot that’s no longer here, just like an LLM output. We have not yet trained a model on human lives yet, just knowledge.

We’re not done with the bitter lesson.