▲ | dragonwriter 5 days ago | |
Structured/constrained generation doesn't protect against outside prompt injection, or protect against the prompt injection causing incorrect use of any facility the system is empowered to use. It can narrow the attack surface for a prompt injection against one stage of an agentic system producing a prompt injection by that stage against another stage of the system, but it doesn’t protect against a prompt injection producing a wrong-but-valid output from the stage where it is directly encountered, producing a cascade of undesired behavior in the system. |