Big context windows are a poor substitute for updating the weights. Its like keeping a journal because your memory is failing.
It reminds me of the movie Memento.