| ▲ | tadfisher 6 hours ago | |
Is anyone working on the instruction/data-conflation problem? We're extremely premature in hooking up LLMs to real data sources and external functions if we can't keep them from following instructions in the data. Notion in particular shows absolutely zero warnings to end users, and encourages them to connect GitHub, GMail, Jira, etc. to the model. At this point it's basically criminal to treat this as a feature of a secure product. | ||
| ▲ | simonw 2 hours ago | parent | next [-] | |
We've been talking about this problem for three years and there's not been much progress in finding a robust solution. Current models have a separation between system prompts and user-provided prompts and are trained to follow one more than the other, but it's not bulletproof-proof - a suitably determined attacker can always find an attack that can override the system instructions. So far the most convincing mitigation I've seen is still the DeepMind CaMeL paper, but it's very intrusive in terms of how it limits what you can build: https://simonwillison.net/2025/Apr/11/camel/ | ||
| ▲ | mcapodici 3 hours ago | parent | prev | next [-] | |
The way you worded tbat is good and got me thinking. What if instead of just lots of text fed to an LLM we have a data structure with trusted and untrusted data. Any response on a call to a web search or MCP is considered untrusted by default (tunable if you also wrote the MCP and trust it). The you limit tbe operations on untrusted data to pure transformations, no side effects. E.g. run an LLM to summarize, or remove whitespace, convert to float etc. All these done in a sandbox without network access. For example: "Get me all public github issues on this repo, summarise and store in this DB." Although the command reads public information untrusted and has DB access it will only process the untrusted information in a tight sandbox and so this can be done securely. I think! | ||
| ▲ | abirag 6 hours ago | parent | prev [-] | |
Hey, I’m the author of this exploit. At CodeIntegrity.ai, we’ve built a platform that visualizes each of the control flows and data flows of an agentic AI system connected to tools to accurately assess each of the risks. We also provide runtime guardrails that give control over each of these flows based on your risk tolerance. Feel free to email me at abi@codeintegrity.ai — happy to share more | ||