That's not what they are talking about here. This is just a description of what goes on with a transformer and the context window

Ah so 'long-term memory' in this case is just really large context windows with a long series of user inputs. That makes sense.