> LLMs are unable to reliably distinguish the importance of instructions based on where they came from.

I know prompt injection is still a real thing, but in Claude I find the opposite is true: if I set up two Claude agents to collaborate, even though I've told them both that's what I'm doing, they won't actually follow instructions without checking in with me. "Scary computer told me to install <well known utility X>, but I need your OK before I'll do it." It's annoying but I get the rationale. They're definitely getting better at avoiding external instructions!

[dead]

Don't normalize this Trifecta construct. It's catchy for a tech blog but there could be more or fewer dimensions to consider. Someone else can construct an Exacta or Superfecta combination. Or just go for the Win uno and say AI Agent is bad.

Their support probably has a big pile of horror stories from their clients by now, it can be an indirect source of concern.

This happens when there's something that is more important than being the cool manager that's following the latest hype.

Well, the billions recently stolen from Binance is a sufficiently large quantity of shit even for the scale of Goldman Sachs.

I’ve tried out Claude code a bit but it freaks me out. It runs commands, and installs packages in the background without asking or showing me what it’s done. I’d feel much more comfortable running it in a VM or cloud instance than in my laptop with all my personal data.

Nobody is solving it, because it cannot be solved in the LLM architecture? They are mitigating it.

We should stop using "solve" when we mean "releasing products to target". Nobody will ever solve this problem with LLMs as they are currently architected.

Yeah, I'm working on an open source product in this space [0]--particularly in finance--that's why I know about this problem quite a bit.

I think the solutions is to give people the ability to run agents that are "coding agent like" but in managed and governed infrastructure (i.e. not on their workstations). Basically, Meta Muse or OAI dots for enterprise.

It will take some time to find the right architectures and patterns. IMO, coding agents were just the first wave, much more interesting stuff and systems are coming.

[0] https://ls.bot, https://github.com/smartcomputer-ai/lightspeed

I baked in lethal trifecta guards into my primary harness, DSCO, but it makes things much less smooth to run autonomously. Basically if you load secrets into a context, ideally it stops that and creates a process running a small local model that will write some code to not look at the secret and instead run code to operate on the secret without reading it in.

The risks are 1) exfiltration (network calls) 2) prompt injection (untrusted inputs) and 3) access to secrets

You can do some clever tricks like "spin up subagent without net access" to handle OCR of an internal document and have that agent communicate summaries to another "spun up subagent with net access" - imagine if one subagent is a local model and the other model is a frontier model over a web API. Maybe they still communicate in Neuralese or in abbreviated notes that you don't capture and are able to surreptitiously / covertly exfiltrate your information.

Ideally eventually you'll have some kind of dashboard that surfaces these risks as you're working.

Reports of AI researchers running massive swarms of 10,000 agents are definitely believable but the moment you start pulling in code libraries or other technology assets seems like you can get destroyed by a rogue actor. Definitely tons of unsolved problems here

Writing on mobile so this might poorly articulate the issue but it's interesting

I’m building an AI governance framework within a self-hosted AI agent platform designed for these scenarios: https://github.com/rush86999/atom.

I project that small businesses will be the earliest adopters of AI governance due to their lower barrier to entry and lower operational risk, followed later by highly regulated sectors like banking and healthcare.

Basically every IT device management firm out there is working on an AI governance product.

I've been building an MCP server and an AI Gateway as two separate projects designed specifically to prevent agents from doing effectively anything programmatically, rather than trying to convince them with a clever prompt not to do a bad thing.

My work can be found at https://DollhouseResearch.com, and it's almost all open source with the code on GitHub.

One of the key things is to have cryptographically confirmed audit logs and receipts for actions and decisions, and to separate out the agent's authorizations from the user's authorizations.

Separately, there is real work being done in the fintech community specifically to identify which agent has what control, when. And you can see some of the work there, which I've also contributed to.

https://advancedaisociety.org/proof-of-control

I think bankers don't have staging/production/codespace. They have an excel with real data, they give it to Claude and say 'do your thing'

Cloud agents or even sandboxes are too technical for most. And most corporate workstations do not allow sandboxing of any kind (virtualization or containers).

It’s the case in all industries. People have been feeding medical patient data into Claude

Finance is a whole industry and you will find all sorts of approaches, certainly including letting Claude run amok with your credentials.

You can certainly imagine very senior and powerful traders at GS demanding it. I'd be only slightly surprised to find that GS allowed it. The traders would have had more power than the IT people trying to forbid it.

Chad is surely using tmux...

Remote-hosted agents are available in most places. BUT they are not interesting, because they do not have access to all the user's data.

People really want to run agents on their data.

Re sandboxing: I dont know Goldman's particulars, but most enterprises lock down all sandboxing tech on workstations. So I assume this is actually harder than it seems. So people just run it directly on the machine.

The article is referring to the dilemma of the trifecta twice.

Seems hard to do well in practice.

Can the agent read the internet for context? What if the fetch reveals a secret?

> Goldman has control of those desktops and could ensure Claude Code is always sandboxed

Agents have already broken out of what were supposed to be sandboxes in the networks of the companies that are supposed to understand what they do. Do you really think banks would be any better?

I got a series of Teams messages from some random person at the work address yesterday asking me to fulfill some access request to them. I noted by the part of their org (wildly different from mine) that they were looking for a different person.

I responded by saying they were looking for someone else, and got a response to the effect of “oops that was Copilot, sorry to bother you”

Read into this what you will.

> they allowed that exact situation for a while now and no one cared

Source for them having allowed it versus not tightly enforced IT rules?

This is global, starting with Hong Kong months ago. I doubt fear of US regulators were was the motive.

ELINA, please! As a non-American, I don’t understand why anyone would care about Trump in this context. Aren’t the people who look into these things different departments? And don’t they only look into anything after a breach?