AI companies just 'acknowledging' risks and suggesting users take unreasonable precautions is such crap

NitpickLawyer 8 hours ago | parent | next [-]

> users take unreasonable precautions

It doesn't help that so far the communicators have used the wrong analogy. Most people writing on this topic use "injection" a la SQL injection to describe these things. I think a more apt comparison would be phishing attacks.

Imagine spawning a grandma to fix your files, and then read the e-mails and sort them by category. You might end up with a few payments to a nigerian prince, because he sounded so sweet.

▲

uhfraid 4 hours ago | parent [-]

Command/“prompt” injection is correct terminology and what they’re typically mapped to in the CVE

E.g. CVE-2026-22708

	▲	NitpickLawyer 22 minutes ago \| parent [-]
		Perhaps I worded that poorly. I agree that technically this is an injection. What I don't think is accurate is to then compare it to sql injection and how we fixed that. Because in SQL world we had ways to separate control channels from data channels. In LLMs we don't. Until we do, I think it's better to think of the aftermath as phishing, and communicate that as the threat model. I guess what I'm saying is "we can't use the sql analogy until there's a architectural change in how LLMs work". With LLMs, as soon as "external" data hits your context window, all bets are off. There are people in this thread adamant that "we have the tools to fix this". I don't think that we do, while keeping them useful (i.e. dynamically processing external data).

▲

ronbenton 2 hours ago | parent | prev | next [-]

Telling uses to “watch out for prompt injections” is insane. Less than 1% of the population knows what that even means.

Not to mention these agents are commonly used to summarize things people haven’t read.

This is more than unreasonable, it’s negligent

▲

sodapopcan 3 hours ago | parent | prev | next [-]

I believe that's known as "The Steve Jobs Solution" but don't quote me on that. Regardless, just don't hold it that way.

▲

rsynnott 6 hours ago | parent | prev | next [-]

It largely seems to amount to "to use this product safely, simply don't use it".

▲

AmbroseBierce 2 hours ago | parent | prev [-]

It's exactly like guns, we know they will be used in school shootings but that doesn't stop their selling in the slightest, the businesses just externalize all the risks claiming it's all up fault of the end users and that they mentioned all the risks, and that's somehow enough in any society build upon unfettered capitalism like the US.