Yep, the only way these things can have "memory" is by shoving previous conversations into the context window.

That's only because we hardcoded their weights in our implementation.

Aside from the cost and slowness, nothing about an LLM prevents feeding recent stimuli in and using it to update the models/retrain.