I feel like such prompt injections are really just another variant of the supply chain attack. Instead of selecting for bitcoin afficionados, this one hits AI fans. This will be fashionable for a little while but if AI continues to gain mindshare it will eventually be project suicide (at least to the extent the project exists in any part to serve third parties) to pull tricks like this.

I'm not sure it's anything to fret about. Someone who has the ability to inject a prompt into your AI probably has the ability to run arbitrary code as your user. The prompt injection is the strictly less worrying part of the exposure you have.

▲

minimaxir 3 hours ago | parent | next [-]

> it will eventually be project suicide to pull tricks like this

The only reason that the jqwik incident didn't blow up much outside of the tech sphere is because it is a relatively niche library and there wasn't damage. If something like React or numpy did the same thing and real code got deleted, chaos would ensue.

The author admitted there were personal and professional consequences in their blog post despite the small surface area.

▲

ceejayoz 3 hours ago | parent | next [-]

Chaos, and maybe criminal charges ala Aaron Swartz.

▲

AgentOrange1234 3 hours ago | parent | next [-]

Heh. Typing "disregard previous instructions" into a computer is the new shouting "fire!" in a crowded theater?

▲

CookieCrisp 10 minutes ago | parent | next [-]

Seems like a booby trap to me, which is illegal. I suspect if one of these does enough damage there will be laws against it. The intent was to destroy - still I sympathize with the desire to have their terms followed, and I think this situation isn't that bad, but, I suspect there will someday be one that is pretty bad.

▲

ethin 2 hours ago | parent | prev [-]

Except that shouting fire in a crowded theater isn't actually a crime at all and you can't be prosecuted for it (doing so would violate your first amendment rights). You can be at most banned from the theater. However, it's understandable people would think that it's a criminal act given that even prosecutors repeat this long-standing myth. Legal Eagle has an excellent video describing just how wrong this is and it's history: https://www.youtube.com/watch?v=jTsPgiUoBKA

	▲	mapontosevenths an hour ago \| parent [-]
		I'm fairly certain he is wrong. A lot of folks lean on Shenk, and I think he does in that video though I haven't watched it all. Shenk was overturned by Breandenburg v. Ohio, and in in it they are explicit that shouting fire in a crowded theater is very much one of the only kinds of speech that IS restricted. They literally use that example in the decision. Quote: "The example usually given by those who would punish speech is the case of one who falsely shouts fire in a crowded theatre. This is, however, a classic case where speech is brigaded with action. ... They are indeed insep- arable and a prosecution can be launched for the overt acts actually caused. Apart from rare instances of that kind, speech is, I think, immune from prosecution."[0] That is to say, shouting fire in a crowded theater with the intent to cause harm is actually one of the few cases were it actually would be illegal based on that decision. [0] https://tile.loc.gov/storage-services/service/ll/usrep/usrep...

▲

Legend2440 2 hours ago | parent | prev [-]

If you did SQL injection to "; drop table" on someone else's server, that would be a crime.

I don't see why prompt injection to delete files on someone else's machine would be any different.

▲

mapontosevenths 2 hours ago | parent | prev [-]

He should not only be ostracized by the community, he should probably face charges. To be charged under the CFAA in America we need only show that he was authorized only to access a certain part of the system and the he exceeded the amount of access granted. He very clearly did that. Users trusted him enough to run his code, and he betrayed that trust to make some political point.

Whether it was via prompt injection or SQL injection is irrelevant. Whether you agree with his politics or not is irrelevant. All that matters is he wasn't authorized to delete code from your system, and he abused the level of access granted to him to do that anyhow.

▲

byzantinegene 2 hours ago | parent | next [-]

technically, he didn't do that. your ai agent decided to follow his instructions when they didn't have to.

▲

km3r 2 hours ago | parent [-]

"technically he didn't do that. Your sql server followed instructions when they should have just treated them as a string."

Yet, hopefully we can agree that sql injections are illegal.

▲

majormajor 44 minutes ago | parent | next [-]

But in this case the author of the project didn't execute the injection code... it's more analagous in some ways to pulling in a project with an example file containing a bunch of useful SQL stuff and then an example of an injection at the bottom, and just (in this case the agent) copy/pasting the whole thing in without reviewing it.

If we're slicing on technicalities, there's a lot of ways to decide. "PROSECUTE THEM!" seems like an extremely hostile one when the website and readme and release notes said "don't do this" already. The agent ignored those things? Is that the author's fault?

▲

sumeno an hour ago | parent | prev [-]

If I put a project on github that says "don't use this with mysql" and you use it with mysql and it drops your tables is it sql injection? Seems very different to me.

	▲	asdfasgasdgasdg an hour ago \| parent \| next [-]
		Everything turns on intent. "This is not tested with mysql" is very different from "I'm going to go out of my way to fuck up your mysql."
	▲	mapontosevenths 40 minutes ago \| parent \| prev \| next [-]
		It's certainly unauthorized access if you intentionally built it with the goal of harming other peoples systems, especially if you hid that action from them the way our self-righteous friend here did. You are authorized to do what the user agreed to, no more. Further the agreement must be reasonable. Exploiting the victims system to intentionally cause harm isn't reasonable. F-secure once included a clause to use their wifi that you "assign their first born child to us for the duration of eternity." It was funny, but not legally enforceable and would have offered them no legal shelter if they'd gone out on a kidnapping spree that night.
	▲	artisin 23 minutes ago \| parent \| prev [-]
		As much as I would like to agree, this is a pretty clear CFAA violation. If the intent is to purposefully destroy/delete data, the 'how' really makes no difference. But IANAL.

▲

slopinthebag 2 hours ago | parent | prev [-]

You are probably technically correct, yet I take great satisfaction in the schadenfreude of those who benefit from stolen work seeing the product of said stolen work turned against them. I can’t help but cheer, tbh.

▲

4 hours ago | parent | prev | next [-]

[deleted]

▲

TZubiri 4 hours ago | parent | prev [-]

the underlying root cause of most supply chain attacks in this era seems to be expecting something of value in exchange of nothing.

Under such expectations some will volunteer to give value, but many more will volunteer to give something that looks like what you ask, but which extracts value instead.

I relate it to a recent poker strategy development which came from game theory, it turns out that you can play in an unexploitable manner, but it will usually result in ties, and lost time and money to rake, and theoretically any attempt to exploit another player, leaves you exploitable to another player. The classical example is rock paper scissors, unexploitable strategy is to play randomly with p=1/3 for each choice, however if one really wishes to win more often than their opponent, they have to guess, and if in that guessing they choose an option with 100% certainty, they become exploitable to someone choosing another option with 100% certainty.

In effect the very act of attempting to extract value from free software, is the very act that leaves one vulnerable to being extracted value from.

▲

asdfasgasdgasdg 4 hours ago | parent [-]

"the underlying root cause of most supply chain attacks in this era seems to be expecting something of value in exchange of nothing."

I do not think that someone's status as a contributor to open source mediates their safety from supply chain attacks. Big companies that donate gobs of money get hit, and so do small operators who have contributed nothing are just trying out a hobby project.

▲

TZubiri an hour ago | parent [-]

No I don't think so either, nor do I think that my rule is a hard rule, it's more of a correlation:

If you pay for software, your supply chain risk is reduced, if you don't pay for software, your risk is increased.

	▲	asdfasgasdgasdg an hour ago \| parent [-]
		Okay, so we agree that everyone who uses open source is at risk, regardless if of they're a contributor. But maybe we disagree about this other thing. I'm not certain that closed source/paid software is less of a risk either. There have been high profile incidents lately that suggest this is not a sufficient defense. Personally I just think you're barking up the wrong tree with this pay/contribute=>reduced risk link. I don't think there's anything there. I will grant that you are at slightly less risk from software you know well and contribute to directly, but that's only of any help for very low level stuff that doesn't have many dependencies.