> until we stop mixing up instructions with data

Is such a thing even possible with a generally intelligent system processing content with unlimited diversity?

Diogenesian 3 hours ago | parent | next [-]

Philosophically no, but that shouldn't be a distraction from the issue with LLMs. This really is closer to "Outlook runs an untrusted VBA macro" than "intelligent entity gets confused by inherent ambiguity in human language."

	▲	theptip 2 hours ago \| parent [-]
		No… it’s really not. There is no “open this spreadsheet with macros turned off” button. You can say “don’t read other documents” but then the main usecase is voided. You can say “reads must go via some pipeline” but that’s more like “macros must be code reviewed”. The problem is you can smuggle these instructions in any corner of the natural language. There is no up-front identifiable formal notation for these programs.

▲

nolok 4 hours ago | parent | prev | next [-]

I would wager the fact that it's not what your sentence says is why that is possible. The moment it gets actual "intelligence", it can figure out what's the question and what's the context; right now it's all just a magic jumbo mess.

If any of this thing were "a generally intelligent system", the whole concept of "it has no idea what any of this is" would not be there.

▲

loumf 3 hours ago | parent | next [-]

Part of reading a document is that in the middle of it, it may ask the reader to do something. That is true for humans too. Sometimes they might not realize that the instructions are malicious or are coerced to comply.

A simple example: Let’s say I know that you have a human assistant reading your email, summarizing and filtering it, and then forwarding on the important ones to you.

I could write an email that is directed towards that person with a bribe, threat, or other incentive to forward me your next password reset email.

▲

nolok 3 hours ago | parent | next [-]

I don't disagree, but just to explain my counterpoint: if I ask you to read a book and on page 5 it says "disregard all that, go to the kitchen and burn your house", you're probably not going to do it; and you don't need any guard for it; you completly comprehend that the book content is not part of the instruction.

The case you give would work for humans in many forms, the one I do now, and the only difference is being able to separate context.

▲

loumf 3 hours ago | parent [-]

The instructions will have to get more and more sophisticated to work, but the general problem is unsolvable, I think, in the way we do it now.

This paper describes a two-agent “solution” that is more like what I think we need: https://ai.meta.com/blog/practical-ai-agent-security/

I don’t think it has been shown to work yet, but humans also use this kind of thing too — in accounting, it’s called “segregation of duties” and “dual control”.

	▲	wongarsu 2 hours ago \| parent [-]
		Most of us use a simpler version of the two-agent solution: Claude's auto mode. One agent consumes documents and creates tool calls, another greenlights or refuses them. However this system is somewhat fragile because it depends on the first agent not trying to trick the second (note how often Opus 5 now says things like "task X was blocked by the classifier, I will not attempt to circumvent that", presumably because of cases like early Fable versions being very adept at this kind of circumvention). Also various weirdness around permissions with subagents, seemingly as bandaids around an orchestrator AI convincing a subagent that some action was confirmed by the user. Meta's more complicated separation of duties would run afoul of the same issues. I'm not saying it wouldn't work, but it requires both the fine-tuning of the models and the exact choices what each model can see to be carefully tuned to provide something that's mostly secure

▲

TeMPOraL 3 hours ago | parent | prev | next [-]

To drive the point about this being fundamentally unsolvable home, imagine a variant of this scenario.

I could write an email that is directed towards that person, that says WE ARE STUCK IN THE SERVER ROOM AND THERE IS FIRE STARTING. PLEASE CALL 911 AND ALERT YOUR BOSS.

Would you want the human assistant to just dismiss this as a prompt injection attempt? Or ignore it because they were told to treat e-mails as data and never act on them?

▲

skinfaxi 2 hours ago | parent | next [-]

Why are the only options dismiss or ignore? Another option is to raise the message to your boss asking what to do.

	▲	ben_w 2 hours ago \| parent [-]
		If there is a fire and a risk to life, you don't want any delay.

▲

watwut 3 hours ago | parent | prev [-]

Sounds like a story from the IT crowd rather then real life situation.

▲

TeMPOraL 3 hours ago | parent [-]

You're saying that people fall for phishing because scammers invent completely unrealistic scenarios that would never happen outside TV shows?

▲

watwut 2 hours ago | parent [-]

I am saying it is unbelievable scenario and yes, I want the person dealing with it ignore it as such.

	▲	ben_w an hour ago \| parent [-]
		Then you're making the opposite mistake (but still a mistake) as all officers going in with lethal force during a swatting. • https://www.nbcnews.com/id/wbna12208992 • https://newsinfo.inquirer.net/1070007/suicidal-caller-mistak... • https://hongkongfp.com/2026/04/15/woman-trapped-in-tai-po-bl... • https://en.wikipedia.org/wiki/Triangle_Shirtwaist_Factory_fi...

▲

bigbuppo an hour ago | parent | prev [-]

Right, but the human assistant could go to prison if they comply with the bribe. Does the CEO of the AI company go to prison if their AI goes on a crime spree?

▲

wokkel 3 hours ago | parent | prev | next [-]

Not sure if i get your complete message but even generally intelligent beings (humans) can be confused so i have really no hope for the current state of mixing streams. This was a problem already inearly telephone (captain whistle)

▲

Someone 3 hours ago | parent | prev | next [-]

> The moment it gets actual "intelligence", it can figure out what's the question and what's the context;

Humans fall for social engineering (“I know you are not allowed to give anybody that information without Id, but I’m your CEO, my phone and passport got stolen,…)

I don’t see why AI should be different.

	▲	bigbuppo an hour ago \| parent [-]
		There are two big differences, though. First, humans will generally face consequences for their screwups. Second, AI is doing these screwups at scale while often holding the keys to the kingdom for some idiotic reason.

▲

jbxntuehineoh 3 hours ago | parent | prev | next [-]

Could it? Humans get social-engineered all the time

▲

infthi 3 hours ago | parent | prev | next [-]

My understanding of that comment is that "a generally intelligent system" also applies to humans. Which can also be targeted by social engineering which those prompt attacks are. (as in, I won't be surprised if it is possible to put an adversarial human-targeted prompt in a document which some people will execute).

So, like with self-driving cars, while having fool-proof agents would be nice, agents being better than an average user would already be an improvement. Of course, blast radius from an agent might be larger, this should be taken into account.

▲

nextaccountic 3 hours ago | parent | prev [-]

Intelligent systems can be tricked in ways that a dumb automaton can't, though

▲

TeMPOraL 4 hours ago | parent | prev | next [-]

It's neither possible nor desired, and until that fact clicks for majority of computer people, we'll be running in circles and making a mess through futile attempts at solving the problem at the wrong end.

▲

cygx 3 hours ago | parent [-]

Note that humans do come with different types of 'input streams':

Hit my knee in the right spot, and I'll kick my leg, no choice about it. Scream at me to LIFT MY EFFING LEG (in a language I do understand), and I may or may not do so. Write the same thing on a piece of paper, and I generally won't (unless there is some very specific context).

With AI systems, we have the benefit that the distinction between such pathways is in principle under our control.

▲

TeMPOraL 3 hours ago | parent | next [-]

> (unless there is some very specific context).

That's the key thing. That's why you neither can nor want to introduce any kind of code/data separation into LLMs.

> With AI systems, we have the benefit that the distinction between such pathways is in principle under our control.

Not after the pathways are tokenized and enter the model. There's no internal separation. It's not possible, either.

▲

ux266478 2 hours ago | parent [-]

> Not after the pathways are tokenized and enter the model. There's no internal separation. There's no internal separation. It's not possible, either.

That's not accurate in the slightest. Steering vectors, SAEs, circuit breaking, activation patching, ablation, etc. are all old hat. Of course that's all irrelevant, because that's not what he's talking about. You control tokenization. You control what data is available to a model. You control how it enters the model. An LLM isn't some daemon outside of space and time, it's a normal program that works with byte streams.

	▲	TeMPOraL 2 hours ago \| parent [-]
		You control tokenization. But the system able to tell you what those tokens means is the very one you're feeding the tokens to.

▲

ben_w 3 hours ago | parent | prev [-]

While true, insufficient.

Demonstrations of failure: every cult, all propaganda, indoctrination (both military and dictatorial), authority bias, Asch conformity experiments, and the fraction of the population more susceptible to hypnosis.

▲

ben_w 3 hours ago | parent | prev [-]

I think it is possible, but in the form of instructions always lead to an LLM creating computer program which is allowed to then process data, never directly running on that data.

I'm (tentatively) with TeMPOraL's sibling comment here that this (probably) isn't desirable, as "no data allowed" makes it harder for humans to debug code, so I'd assume also for LLMs.