What stops the agent from just writing code to actually read the secret? That’s kind of the core of the problem. The agent can be tricked into doing anything.

▲

arthurcolle 11 hours ago | parent | next [-]

Indeed. But if it's a model running on your own hardware it isn't as risky. And the Anthropic models are actually pretty good at kind of writing code that will look at contents of a file without fully looking at the content, so it's insecure but not quite as bad as an open("secrets.txt").read()

	▲	cedws 9 hours ago \| parent \| next [-]
		Local model or not, all LLMs are susceptible to prompt injection and making mistakes (like those cases where you see the agent randomly deleted someone's home dir.) That's what Goldman are likely worried about. Not Anthropic going rogue or something. Concealing API keys is a relatively minor protection in the wider picture of the risks.
	▲	spockz 10 hours ago \| parent \| prev \| next [-]
		What does that mean? That it only reads part of it into context or RAM?
	▲	heartbreak 6 hours ago \| parent \| prev [-]
		What does using a local model have to do with it? I’m not worried about the secrets going to Bob, I’m worried about them going to Eve.

▲

solyin 10 hours ago | parent | prev | next [-]

I don't know about GP's setup, but you give the AI an MCP that can do the thing without exposing the key to the AI directly. An all powerful agent could setup a fake endpoint and send itself requests and exfiltrate the key that way, so don't let it do that.

▲

tha_infra_guy 11 hours ago | parent | prev [-]

I wil encourage you to check https://runeward.io this is specifically the usecase we are building this for