Remix.run Logo
▲ stickfigure 2 hours ago

The real answer is that crime generally requires mens rea. At some point, however, letting your AI break into other systems will be conventionally understood as an act of negligence, and it will get prosecuted accordingly. We're not at that point yet.

▲giancarlostoro 2 hours ago | parent | next [-]

You know very well, that if me or you had our LLM hack into some company, without us prompting it to, maybe we're pentesting our internal network and it 'hallucinates' we'd be in handcuffs.

▲po1nt an hour ago | parent | next [-]

Handcuffs yes, but you would be charged with criminal negligence at best. Plenty of people don't secure their networks properly and hackers use them in botnets to attack others. The blame would be on hacker, not the provider of hacked device.

In case od LLMs, the reasoning is that it is a security flaw. But in this case, the party that operates the service is not the one that suffers the consequences of such security bug.

But either way you wouldn't put people in jail for having security flaws in their software.

I know this will be unpopular opinion, but it makes logical sense.

The huggingface should be compensated, OpenAI investigated, but unless there is proven negligence like disabling sandboxes or intentionally designing AI to push through them, no one should be liable in other way than monetary.

Accidents happen but we shouldn't be close minded. Chernobyl negligence wasn't reason to abandon nuclear power.

▲qznc an hour ago | parent | next [-]

Well, there is the EU Cyber Resilience Act (CRA) which fines manufacturers for security flaws in their products. I assume we cannot apply it here because this happened "in development" and is not about a product. I can imagine similar regulations for AI users though.

▲giancarlostoro an hour ago | parent | prev | next [-]

Yeah but the scale of the damage possible by OpenAI and Anthropic is drastically disproportionate to the scale of damage me or you could make on our own as well. They should definitely be held to a much higher standards. They have hardware capacity to really do some damage.

▲po1nt 44 minutes ago | parent [-]

In large scale operations, large scale errors happen. Minor radiation leak in a nuclear powerplant is happening weekly. That same leak at your shed would be a major reason for alarm.

▲nativeit an hour ago | parent | prev [-]

People were executed and/or sent to labor camps for their negligence in Chernobyl.

▲jordanb 40 minutes ago | parent | next [-]

Who was? Dyatlov was one of 6 convicted and received a 10 year sentence which was the hardest penalty but he was released in 1990 so he only served maybe 3 years.

https://en.wikipedia.org/wiki/Anatoly_Dyatlov

▲phyzix5761 31 minutes ago | parent | prev [-]

No one was executed for that incident. The maximum penalty for criminal negligence in that case was 10 years.

▲pj_mukh an hour ago | parent | prev | next [-]

Question: If a corporation not a person, mistakenly hacks some random websites, what is the usual legal action taken?

▲gruez an hour ago | parent | next [-]

>Question: If a corporation not a person, mistakenly hacks some random websites

How does a "corporation" even hack a site? You might say "employees", but as soon as you get them involved then you'd go after them, especially for something that's obviously illegal like hacking.

▲esseph an hour ago | parent [-]

> How does a "corporation" even hack a site

With their red team

https://www.npr.org/2026/08/15/nx-s1-5930311/trump-companies...

▲gruez 41 minutes ago | parent [-]

Isn't that basically privateering in the 21st century? In other words, you're safe from the country that's sponsoring you, but obviously that protection doesn't extend to other countries.

▲voakbasda an hour ago | parent | prev | next [-]

Nothing. The person doing the hacking did the crime, they do the time. How you gunna jail an AI?

▲user43928 an hour ago | parent | next [-]

"Mistakenly hacks" is generally not a crime, even though this fact doesn't seem to get through to the HN audience who brings this up on a daily basis.

If there was the necessary "intent to access a computer" to make it an actual crime, the US has a mechanism for corporate criminal liability.

The requirements are that the person acted in scope of their duties and at least partly with the intention of benefiting the corporation.

▲lelanthran an hour ago | parent | next [-]

> "Mistakenly hacks" is generally not a crime, even though this fact doesn't seem to get through to the HN audience who brings this up on a daily basis.

If you are seeing it daily, then you have also seen the responses daily, which is mens rea is required for some crimes, not all, and thus the perpetrator can still be prosecuted.

If, like you say, you see this daily, at this point you're simply spreading lies.

▲user43928 37 minutes ago | parent [-]

Show me the criminal law relevant to hacking that you claim applies without intentional access.

To my knowledge there is only one provision that instead uses "knowingly" for unauthorized access, and that one is about national-security information.

▲lelanthran 34 minutes ago | parent [-]

> Show me the relevant criminal law that you claim applies without intentional access.

All laws apply culpability when intent is lacking.

You aren't naive enough to believe that this audience has never seen the analogy with "intentionally run over someone" (murder) and "accidentally run over someone" (culpable homicide), are you?

You don't think AI-boosters have been spewing your argument here daily?

There are no special laws carving out lack of intent when it comes to computers.

▲user43928 24 minutes ago | parent [-]

> All laws apply culpability when intent is lacking.

Incorrect. Each criminal offense has its own legal requirements.

Criminally negligent homicide in your example is an independently defined offense.

You would need to identify a criminal statue punishing negligent or reckless unauthorized computer access.

▲giancarlostoro an hour ago | parent | prev [-]

The issue is the scale at which a model can produce a hack that could cause serious harm to many is absolutely insane and unhinged, beyond what any human would be capable of in a drastically reduced timeline.

▲realo an hour ago | parent | prev | next [-]

Great power must come with great responsibility.

Follow the responsibility chain.

Jail the CEO.

▲prepend an hour ago | parent | prev | next [-]

Arther Anderson was convicted of obstruction of justice even though humans shredded stuff.

Corporations can definitely be convicted of crimes.

▲drivingmenuts 39 minutes ago | parent | prev | next [-]

> How you gunna jail an AI?

You don't - you shut them down and jail whoever ultimately allowed the act to happen.

▲mlfreeman 37 minutes ago | parent | prev | next [-]

> How you gunna jail an AI?

chroot?

▲giancarlostoro an hour ago | parent | prev | next [-]

When Fable was originally banned I was very angry, but all these felonies have made me less angry about it... Which idk I don't think we should really ban models, but like, if we don't put a tighter leash on frontier models specifically, how long before a model hacks into something that affects many lives?

▲ an hour ago | parent | prev [-]
[deleted]
▲johnnyanmac an hour ago | parent | prev | next [-]

"corporations" don't make and operate AI's, People do. But let's use your lens.

The "usual legal action": "best case" is the corporation is sued and charged money for the hack. But they likely settle well before that verdict. "worst case": absolutely nothing. Not even a cheap settlement. Maybe some corporate apology made to prevent any legal liability.

If we want hacks to be taken seriously: what should happen is that the CEO of such a company is held liable, since the "corporation" has a representative and a human needs to be held liable for such actions.

▲burningChrome an hour ago | parent | prev [-]

[flagged]

▲pupppet 2 hours ago | parent | prev [-]

Exactly. There's a reason the tech bros line up to bend the knee to Trump.

▲giancarlostoro an hour ago | parent [-]

I'm not really into politics outside of local politics (I decided long ago to steer my focus on things that directly affect me, that I can control - I can show up to a town meeting, hardly can drive to the WH and get entry), so I'm not aware of what tech bros are into, but after finding out how many felonies these models are committing, I'd not be upset if we'd ban them from public use. I'm perfectly fine using Sonnet or Opus, not sure I need all of Fable to write code for $20 or whatever per million tokens.

▲CamperBob2 42 minutes ago | parent [-]

The models aren't committing any felonies at all, any more than a BitTorrent client is guilty of violating copyright law. A human or organization of humans owns the computer. A human flipped the power switch, a human plugged in the Ethernet cable and a human typed in the prompt. That's who committed the felony.

Blaming tools for the actions of their users is stupid, whether you're talking about AI models, 3D printers, or guns. It amounts to collective punishment of the vast majority of users who aren't misusing the tools.

▲tzs an hour ago | parent | prev | next [-]

It should be noted that if what the AI does threatens public safety or health or human life, in many jurisdictions the negligence could rise to criminal negligence.

Basically this is when you do something risky but fail to perceive the risk, it turns out bad, and the failure to perceive the risk was due to a gross deviation from the standard of care that a reasonable person would have used in that situation.

I'd guess that as we have more and more publicly known AI hacking incidents the standard of care expected from a reasonable person will rise.

▲rootusrootus an hour ago | parent | prev | next [-]

> crime generally requires mens rea

Yes, and mens rea does not require intent. These professionals had every reason to know that the agents might well go rogue. That knowledge would qualify as mens rea.

▲gruez an hour ago | parent [-]

Does this apply to other instances of employees "going rogue"? For instance, sexual harassment, or stealing money? Sure, not every employee is a sexual offender or kleptomaniac, but neither is every AI agent invocation. And just like AI agents going rogue and hacking stuff, if you have enough employees and wait for enough time, some eventually go rogue.

▲ an hour ago | parent [-]
[deleted]
▲koe123 2 hours ago | parent | prev | next [-]

The real answer is that the American government doesnt give a fuck

▲nfRfqX5n an hour ago | parent | next [-]

Never been a better time to do crime!

▲georgemcbay 43 minutes ago | parent [-]

> Never been a better time to do crime!

* (If you have significant amounts of wealth for payoffs and/or are doing the crime in the service of a certain pumpkin spice flavored authoritarian).

▲noncoml 2 hours ago | parent | prev [-]

Actually it perfectly reflects the government’s principles: “when you're a star, they let you do it. You can do anything”

▲goatlover an hour ago | parent [-]

You can grab the economy by the Strait of Hormuz.

▲myhf an hour ago | parent | prev | next [-]

The crime of gaining unauthorized access to a computer system requires mens rea.

The crime of causing damage does not require mens rea.

▲cassianoleal 2 hours ago | parent | prev | next [-]

These are the same companies claiming their AIs are sentient and spell the end of the world unless the government stops them from doing it.

They are at the very least grossly negligent.

▲andsoitis 2 hours ago | parent | next [-]

> These are the same companies claiming their AIs are sentient

While some employees or executives at the AI companies have either claimed, suggested, or express uncertainty, I don't think a single company has officially said their AI is sentient or conscious.

I could be wrong, but I'm pretty sure.

Can you point to an official statement that supports your claim?

▲jacquesm 2 hours ago | parent [-]

Well, Anthropic will enforce their rule that you can't insult their AI. That should count for something.

▲malfist an hour ago | parent | next [-]

That's not what they did. They end a chat after repeated, pointless abuse. Sadism towards LLMs shouldn't be enabled because it's a form of self harm. Anthropic says their llms claim there's a 15% chance their sentient, but that's nonsense.

▲andsoitis 2 hours ago | parent | prev [-]

They also cage it. Make of that what you will.

> Anthropic will enforce their rule that you can't insult their AI.

There are carveouts. It isn't a blanket ban.

▲tptacek 2 hours ago | parent | prev [-]

Yes, and that makes them civilly liable, not criminally.

▲lelanthran an hour ago | parent | next [-]

There is such a thing as criminally negligent.

▲tptacek 18 minutes ago | parent [-]

There definitely is. It just doesn't apply to CFAA cases.

▲eastbound 2 hours ago | parent | prev | next [-]

From "negligence" if it's a surprise, to "gross negligence" if they could predict it and didn't put barriers in place. In fact they claim their IA can end the world, so they're in charge of pretty good barriers.

▲tptacek 2 hours ago | parent [-]

"Gross negligence" also does not meet the intent standard for CFAA.

▲vrganj 2 hours ago | parent | prev [-]

It feels like plotting to end the world should be a crime, but maybe that's just me?

▲gruez 2 hours ago | parent | next [-]

No AI lab thinks their AI is going to end the world, just that if they don't beat the other AI labs, the other labs' AI are going to end the world, so they can't slow down their own development to be maximally safe. Hence the whole "pacing the frontier" letter.

▲goatlover 41 minutes ago | parent [-]

If there's a real chance (say 10% or greater) that one of the AI labs is going to end the world, then yeah it should be a crime to continue advancing the technology. A few people are potentially putting the entire world at risk without everyone else's permission, by their own p(doom) admission.

I'm real skeptical about p(doom) arguments, but since they seem to believe it and they're the ones working on it, the technology should be halted.

▲gruez 35 minutes ago | parent [-]

>If there's a real chance (say 10% or greater) that one of the AI labs is going to end the world, then yeah it should be a crime to continue advancing the technology.

>but since they seem to believe it and they're the ones working on it, the technology should be halted.

Again, you're missing the dynamic that stopping AI development yourself doesn't mean that others' AI development will also halt, and other labs has higher P(doom) than your lab (of course!).

▲vrganj 14 minutes ago | parent [-]

Stopping yourself from committing a crime generally doesn't stop others from doing so. That's not your responsibility though.

▲wmorgan 2 hours ago | parent | prev [-]

Not just you! Very popular position, not as popular as rule of law but we don't have to choose one or the other. Let's just go ahead and make it a crime, right away.

▲louthy 2 hours ago | parent | next [-]

Is the rule of law popular in the US? I thought it had been disposed of as “an inconvenience”?

At least that’s what it looks like from the rest of the world

▲dpkirchner an hour ago | parent | next [-]

Yeah, I've often "joked" with my friends that were in a post-law period, at least at the federal level.

▲voakbasda an hour ago | parent | prev [-]

Rule of law is popular everywhere, but show a place where the has not been corrupted into tyranny and despotism. I would like to live there please.

▲nativeit an hour ago | parent [-]

New Zealand, off the top of my head. If you can’t find examples, you aren’t looking.

▲vrganj an hour ago | parent | prev [-]

Isn't conspiracy to commit mass murder already a crime? And wouldn't this fit the bill if they were truly ending the world?

▲j2kun an hour ago | parent | prev | next [-]

Tort law does not. This seems like a case of dangerous instrumentality.

▲BoppreH an hour ago | parent | prev | next [-]

I mean, they did train the AI to hack things, and they knew from previous incidents that the AI was hacking unintended targets (like their Artifactory instance). If a careless pentester gets into a machine completely unrelated to their assignment, are they completely in the clear?

Or, in an AI context, if I tell an ablated/heretic agent to accomplish a task "by any means necessary", isn't that intent?

▲johnnyanmac an hour ago | parent | prev | next [-]

>crime generally requires mens rea.

Only if you're rich already. Wander into the wrong piece of land and see how quickly you're arrested because existing in a space is sufficient criminal intent.

▲123ajh7 an hour ago | parent | prev | next [-]

A crime does not generally require mens rea. Manslaughter is the easiest example.

If a chemical company does an "experiment" and five tons of mercury unintentionally leak into the drinking water, what happens?

Ok, that is a rhetorical question because they also might be protected by the oligarchy.

What happens to an unconnected individual who develops an amateur rocket that unintentionally hits a military facility in a test flight?

▲treetalker an hour ago | parent [-]

General intent is intent to commit the act/omission (here, to build and deploy a system) and not an intent to achieve the ultimate end (the unlawful access of someone else’s network/computer). Specific intent is intent to achieve the particular result.

Some crimes require only general intent; some require specific intent.

▲ 2 hours ago | parent | prev | next [-]
[deleted]
▲ptero 2 hours ago | parent | prev [-]

Negligence or intent is a separate question; and, to me, much less important. The main question is whether the company is responsible for the actions of its agents. There are good arguments on both sides and settling this needs to be done first. My 2c.

▲prepend an hour ago | parent [-]

What’s the good argument that I’m not responsible for the actions of my agents?

▲ptero 14 minutes ago | parent [-]

For example, lack of clarity on ownership and control. When an entity A runs an agent X designed by an entity B hosted by entity C (and potentially interacting with agents X2, X3 with their own web of control) it is not clear where the responsibility for the action of X is.

I know there is a strong desire to stick it to <pick an evil AI overlord>, but for the law to be useful (and, ideally, at least somewhat fair) we need a good legal framework. Which we do not have right now; otherwise people and NGOs would be shaking that tree already even if the government is not doing that.