Prompt injections are a whole class of vulnerabilities, and I would say there is generally a pretty good idea of how to mitigate them to be impactful. However in many cases those mitigations are not implemented (in the strictness that they require), as they are usually either too costly (second LLM as judge) or lead to worse UX (tool call confirmation with appropriate review of all input parameters on every tool call; disconnecting web access).

▲

xienze 4 hours ago | parent [-]

> and I would say there is generally a pretty good idea of how to mitigate them to be impactful

Yes and no. No in the sense that the space of possible ways to craft a malicious prompt is infinite. Yes in the sense that you can lock down every single possible way the agent can interact with the system. But, will doing so render the agent nearly useless? And, are you absolutely sure you'll never forget to lock each and every thing down, including things you weren't aware of?

> second LLM as judge

Again, see above. You're perhaps making it harder to craft a prompt injection, but not impossible. This is a false sense of security.

	▲	hobofan 3 hours ago \| parent [-]
		It very much depends on what kind of system you are building, as each of them have different input/output modalities, each combination of them allowing for different attacks. If you are building a generic agent that can theoretically connect to anything and should build things end-to-end without interventions, then yes, it's very intractable to defend against prompt injection. In more narrow cases, like Chat UIs it becomes a lot easier, though if it should appeal to a generic audience, still easy for individual users to misconfigure. And if you want to use it in the most high-security environments where nothing can leak in/out, you will have to air-gap the system anyways (like any traditional software). > You're perhaps making it harder to craft a prompt injection, but not impossible. This is a false sense of security. It's not a false sense security, it's part of a layered security strategy. Yes, it will never be impossible, but so are many individual steps in cybersecurity attacks. There are other systems (like email) that are essentially impossible to fully lock down with purely mechanical security measures if you want to allow for meaningful work (e.g. having email attachments). A second-judge LLM when paired with keyword/pattern blocklists, and active alterting/lockout after repeated attack attempts can form a very robust line of defense that in practice can be near-impossible to break. For many attacks, to have actual exploitability, you also need to have compromised a peripheral system (or user account) to have repeated attempts at circumventing prompt injection measures. > And, are you absolutely sure you'll never forget to lock each and every thing down, including things you weren't aware of? That's part of every normal (non-LLM) security audit. If you don't know what data can potentially go where, then you are open in attacks in any system. The AI space does add a bit of complexity here, if using MCPs hosted with third parties, though.