When firing up agent VMs for my personal projects, I tried to completely separate SSH keys as well. Surprisingly annoying to have multiple keys for, say GitHub, and restrict the agent to use a deploy key.

I saw this project enables ssh-agent forwarding, so my question; is this a non-issue to begin with? Or just not your focus currently.