Is this issue only about the memory? Wouldn't it be possible to have it expose any information that it currently has like current project information, code, credentials etc?

Sharing those things to coding agents and model providers is probably inevitable for the use. The memory with random tidbits is not.