If done correctly, the agent will only be able to request that passwords be filled, but the secrets will only be handled by the password manager and browser extension, which will be able to set some security boundaries and make checks on which domains are allowed.

If your issue is allowing agents to use your credentials and take actions on your behalf, then that is a separate question.