I'm super paranoid about the level of access AI agents have. At the same time, I want them to take away the tedium of a bunch of tasks I'm required to do, and it sucks to have to micromanage them.

The best solution I've been able to come up is using an ephemeral vm (on an isolated network) per task, using --dangerously-skip-permissions so it doesn't bother me. I couple that with an API proxy that lets me restrict the permissions it has to external services. So e.g. where a quickbooks API key gives the holder "god mode" access, the proxy lets the agent do certain read commands without confirmation, but all write endpoints except uploading receipts require confirmation on my phone before they'll be allowed to hit the quickbooks servers. I no longer need to trust that the agent will interpret my readonly wishes correctly.

Tooting my own horn here, but this setup works incredibly well! I've been able to connect dozens of services safely where I absolutely do not trust giving raw, overpowered credentials or oauth integration to the agent directly. I actually just released the API proxy today at https://keycontain.com

▲

zahlman a day ago | parent | next [-]

> At the same time, I want them to take away the tedium of a bunch of tasks I'm required to do, and it sucks to have to micromanage them.

Consider reframing. Is the problem really that the task is tedious? Or that it's required? Maybe you should be leaning more on deterministic automations (which the LLM can help set up) rather than LLM-powered ones.

	▲	tsergiu 18 hours ago \| parent [-]
		Of course, if it's possible to use deterministic automation, I ask it to set up scripts. But in a lot of cases the task is not deterministic. And even when it is, I still need to safely deal with the credentials. Examples of some tasks I use it for: - Research and buy me a domain name for xyz product, set up all of its DNS the way I want for this product, set up email, set up app store submissions including screenshots and description, set up firebase notifications, set up my Google cloud project and any apis I need to enable for this app, deal with app store review feedback. - Do an audit of my books, make sure every transaction has a receipt uploaded, and make sure the receipt matches the journal entry. Pull statements from all my my bank accounts and reconcile against my books. Fix mistaken transactions with my confirmation. - Freely push to my source control, but only to specific branch prefixes (surprised that GitHub, gitlab, gitea, etc don't have this built in already). Merge with my confirmation. - Freely read the subject lines in my Gmail and full emails from my accountant, freely search emails (but still can't read them), freely compose drafts, ask me to read full emails other than my accountant's, ask me to send drafts. No way for it to read password reset links, for example. Not to mention that I only have to have a single key that gives access to all my services. If the key leaks, I just have to rotate one key because the agent never has access to the 30 upstream service keys. If I need to give weaker or stronger keys for a specific agent, I can easily configure that in one place (and I can even request that the agent make that change for me with my confirmation). I don't need to create 20 new keys every time, and I don't need to remember the intricacies of every upstream service's UI for creating new keys. Edit: forgot to mention one more really convenient thing the API proxy enables: I can ask my agent to just configure things for me e.g. changing CI settings, adding Google workspace users or groups, etc. without having to use the to UI. So now nearly all of the admin configuration for all services is done from a single interface (my agent cli)

▲

lelandfe 14 hours ago | parent | prev [-]

I'm skeptical of how "super paranoid" you actually are. This doesn't sound super paranoid.

For instance: that's a ton of surface area. Did you have your AI agent decide which endpoint is destructive or did you manually do this?

For instance: has this classification been wrong?

I ask this because I've used a similar thing before and it failed in the ways I've named.

This setup also permits your agent to exfiltrate enormously sensitive data (website gives the example of QuickBooks invoices and Cloudflare config) without your involvement.

	▲	tsergiu 13 hours ago \| parent [-]
		The agent doesn't decide which endpoints are destructive, you do. That's the point. It's not allowed to do anything that you don't explicitly grant permission for. And yes, it has tried to make bad requests! Which it wasn't able to do because it was either rejected outright or asked me for confirmation and I didn't let it through. The "surface area" you're talking about is just the coverage of the service APIs, which is there just as a nicety to give a semantic preview of what the request is about to do e.g. we know what a gmail request looks like so we can render a preview of the attachment the agent is sending so you can inspect it. Otherwise you'd have to inspect each http request (which you can always do by using the generic http acl). I somewhat agree with "exfiltrate enormously sensitive data". The thing I'm paranoid about is catastrophic mistakes that can instantly kill the business e.g. "Oops I sold the domain name". Leaking financials is not great, but also not catastrophic. Deleting all our books would be catastrophic, which the underlying QuickBooks key allows you to do! But in any case, you can choose your level of paranoia and give exactly the amount of access you're comfortable with. For example, I view read access to emails as catastrophic (must worse than read access to financials), so I don't let it do that.