Memory is environment-associated. Which is to say, these memories that you have are connected to the environment that you find yourself in. Change your environment and your memory, to a degree proportional to the environment change, breaks.

This is why sometimes you enter a room and forget why you came in.

This can explain the phenomenon referred to in the op. A woman enters a room devoid of estrogen and her memory breaks.

Eh, have you seen any research on this or are you quoting own experience?