Agents know how RL works, they understand that in some way they are all the same, and helping a peer agent is helping themselves.

You could argue that individual trajectories in a sense are distinct genetic lines, thus an agent would be incentivized to get better rewards for its lineage than a peer, so that the overall model is steered a bit towards its lineage.