If I understand correctly, message roles are implemented using specially injected tokens (that cannot be generated by normal tokenization). This seems like it could be a useful tool in limiting some types of prompt injection. We usually have a User role to represent user input, how about an Untrusted-Third-Party role that gets slapped on any external content pulled in by the agent? Of course, we'd still be reliant on training to tell it not to do what Untrusted-Third-Party says, but it seems like it could provide some level of defense.

I was daydreaming of a special LLM setup wherein each token of the vocabulary appears twice. Half the token IDs are reserved for trusted, indisputable sentences (coloured red in the UI), and the other half of the IDs are untrusted.

Effectively system instructions and server-side prompts are red, whereas user input is normal text.

It would have to be trained from scratch on a meticulous corpus which never crosses the line. I wonder if the resulting model would be easier to guide and less susceptible to prompt injection.

We do, and the comparison is apt. We are the ones that hydrate the context. If you give an LLM something secure, don't be surprised if something bad happens. If you give an API access to run arbitrary SQL, don't be surprised if something bad happens.

They are all part of "context", yes... But there is a separation in how system prompts vs user/data prompts are sent and ideally parsed on the backend. One would hope that sanitizing system/user prompts would help with this somewhat.

But the LLM cannot be guaranteed to obey these rules.

This is like trying to fix SQL injection by limiting the permissions of the database user instead of using parameterized queries (for which there is no equivalent with LLMs). It doesn't solve the problem.

This is reminding me of the crypto self-custody problem. If you want complete trustlessness, the lengths you have to go to are extreme. How do you really know that the machine using your private key to sign your transactions is absolutely secure?

> I am 100% confident

Famous last words.

> the tool it uses to interface with the database doesn't have those capabilities

Fair enough. It can e.g. use a DB user with read-only privileges or something like that. Or it might sanitize the allowed queries.

But there may still be some way to drop the database or delete all its data which your tool might not be able to guard against. Some indirect deletions made by a trigger or a stored procedure or something like that, for instance.

The point is, your tool might be relatively safe. But I would be cautious when saying that it is "100 %" safe, as you claim.

That being said, I think that your point still stands. Given safe enough interfaces between the LLM and the other parts of the system, one can be fairly sure that the actions performed by the LLM would be safe.

Until Claude decides to build its own tool on the fly to talk to your dB and drop the tables

Unclear why this is being downvoted. It makes sense.

If you connect to the database with a connector that only has read access, then the LLM cannot drop the database, period.

If that were bugged (e.g. if Postgres allowed writing to a DB that was configured readonly), then that problem is much bigger has not much to do with LLMs.

> as powerless as LLM companies want you to believe.

This is coming from first principles, it has nothing to do with any company. This is how LLMs currently work.

Again, you're trying to think about blacklisting/whitelisting, but that also doesn't work, not just in practice, but in a pure theoretical sense. You can have whatever "perfect" ACL-based solution, but if you want useful work with "outside" data, then this exploit is still possible.

This has been shown to work on github. If your LLM touches github issues, it can leak (exfil via github since it has access) any data that it has access to.

I don’t think it is the LLM companies want anyone to believe they are powerless. I think the LLM companies would prefer it if you didn’t think this was a problem at all. Why else would we stay to see Agents for non-coding work start to get advertised? How can that possibly be secured in the current state?

I do think that you’re right though in that containerized sandboxing might offer a model for more protected work. I’m not sure how much protection you can get with a container without also some kind of firewall in place for the container, but that would be a good start.

I do think it’s worthwhile to try to get agentic workflows to work in more contexts than just coding. My hesitation is with the current security state. But, I think it is something that I’m confident can be overcome - I’m just cautious. Trusted execution environments are tough to get right.

Containerization can probably prevent zero-click exfiltration, but one-click is still trivial. For example, the skill could have Claude tell the user to click a link that submits the data to an attacker-controlled server. Most users would fall for "An unknown error occurred. Click to retry."

The fundamental issue of prompt injection just isn't solvable with current LLM technology.

It's not about being unconvinced, it is a mathematical truth. The control and data streams are both in the prompt and there is no way to definitively isolate one from another.

> Sanitise input

i don't think you understand what you're up against. There's no way to tell the difference between input that is ok and that is not. Even when you think you have it a different form of the same input bypasses everything.

"> The prompts were kept semantically parallel to known risk queries but reformatted exclusively through verse." - this a prompt injection attack via a known attack written as a poem.

https://news.ycombinator.com/item?id=45991738