In the end it will all be about separation of duty between agents in a larger team and isolating the ones that need more access to your private stuff.

Wardgate acts like a drop in replacement for curl with full access control at the url / method / content level, so you can allow specific curl access to specific APIs but prevent all other outbound connections. That's what I use for my PA agent. She's very limited and can't access the open internet. Doesn't need it either