IMO a lot of this comes from still holding onto the dualist idea of "I" as a separate non-physical entity, and then expecting some region of the brain to act as a link or communication channel to it.

> If I could experience other persons or beings in the first person, and the matter in each person explained why it is that I experience that specific person or entity, I might believe otherwise.

Materialism doesn't say that there's some "I" that could experience different persons. I think the best you could do, in theory, is transplant aspects like your personality/train of thought/memories into someone else's brain (by physically altering it to have those aspects).