Keep in mind that not every agent is so naive as to allow a local client to connect to it without reauthenticating somehow.

1Password, for example, will, for each new application, pop up a fingerprint request on my Mac before handling the connection request and allow additional requests for a configurable period of time -- and, by default, it will lock the agent when you lock your machine. It will also request authentication before allowing any new process to make the first connection. See e.g. https://developer.1password.com/docs/ssh/agent/security