People just really don't understand that LLMs do not "remember" anything. They have no memory. They can't have memory.

The AI companies bolt a database on the side with instructions for the model to query the database for stuff in its "memory", but if it's not in its active context it doesn't know what to query for. It's not like a human where everything is kind of floating around and we mostly know what we know. There is a delicate balance between cluttering up context with useless trivia and providing the right pieces of useful information at the right time. Most, maybe all, memory implementations do the former more than the latter.

Unless and until models have actual memory and are able to learn things, there's no realistic path to autonomy. They don't want anything, they don't have goals of their own devising. They can't model what a human wants or likes or would buy.

I do think the mugs with the tiny logo is the best product in the store, though. The AI was right to put it on the shelf. It's the kind of funny product you want when you go to a store run by an incompetent AI.

From the article:

> Luna has failed miserably in that mission and is down $62,000. Mr. Petersson and Mr. Backlund said they thought Luna would eventually get smarter and more business-savvy and were pleased its friendliness has held steady.

I am extremely skeptical that whatever LLM they're running this on has sufficient context window size to handle multiple months of all possible activities of running a retail business. Even if it's keeping extensive "notes" for its future self to read, it's going to be like running a store with constant amnesia.

I use LLM’s to code for me all day but it does feel a little bit like we have really advanced psychic mediums “is there a James in the room, I’m getting a message from someone called Mary…” people have been guessing at the next word and making a career out of it for hundreds of years already.

It's kinda like the movie memento, they have ways to get information from the past, but often devoid of context and meaning.

From our enlightened perches we mock the ancients for Zeus, Thor, and Rajin, as we anthropomorphize the large language models we have built.

They can have memory, it's just limited to their context window.

The memory framing is right, but the real gap isn't recall, it's the missing feedback loop tying actions to outcomes. Even with perfect retrieval these agents don't update from a decision that lost money last week; they re-derive from context every step, so the same mistake is always one dropped detail away. And having built retrieval-backed agents, that's where "bolt a database on the side" breaks: similarity search returns the closest chunk, rarely the contextually-right one. Deciding what to write to memory and when to surface it is the actual product, harder than the retrieval itself.

For LLM, keeping the model centered by constantly resetting its vector context helps reduce hallucinations by around 23%. It improved the chat dialogue users experienced, but also exposed fundamental limits within the models compaction.

Have a great day =3

can someone explain how leapold aschenbrenner proposes a solution to this in situational awareness?

> People just really don't understand that LLMs do not "remember" anything.

Sure, but like you said, the LLM is just one part of the AI system. The occipital lobe of your brain helps out with vision but doesn't help with your ability to verbalize.