Agents and harnesses don't get access to "my" ssh keys, they get access to new ed25519 key pairs created for specific projects and access to discrete things. The blast radius is relatively well contained to specific VMs they are SSHing into for project specific purposes. I don't run a harness or agent directly on my personal workstation.