You can add a tag to each memory indicating who wrote it, and then at read time filter the memories instead of putting them in the prompt.

I am the founder of Substrate, a project that lets you inspect what is in the memory and select what each agent sees.

Do you separate memory per user with your harness?

Yes we do. We separate based on both the end user identity and agent identity. Would love to see how your product could potentially work with alter