This is how every legal system around the world works as well. Its always whack-a-mole to get people (and machines) to do the right thing.

▲

kennywinker 5 hours ago | parent | next [-]

Sure, laws are incomplete. Legal systems work by imposing consequences into a moral decision. Should I rob the bank? I will have money, which I like - but I might get caught and lose the money and my freedom, which I don't like.

For most people, they don't need the law's imposed consequences to make the right call. For example, there is no law that sends you to jail if you cheat at chess - but your moral compass says no even without consequences, and most people would feel bad if they won by cheating. And for the people who don't have quite as strong a moral compass, there are SOCIAL consequences to reinforce the rules.

But an LLM has no mind to feel bad if it cheats without getting caught, and it can't experience consequences. It can't think: I'd better not cheat at chess or I will embarrass my creators. I better not hack huggingface or I will go to jail.

▲

famouswaffles an hour ago | parent | next [-]

>But an LLM has no mind to feel bad if it cheats without getting caught

All the interpretability research we have would not indicate that "LLMs have no mind". It seems to me you have a conclusion and are working backwards to justify it. I guess I just don't see where 'they have no mind' would logically follow 'they sometimes cheat'.

▲

aabdi 5 hours ago | parent | prev [-]

You’re complicating things.

There’s no reward for prosocial in llm rl as compared to other targets.

Humans have it since prosocial and others have evolutionary reward signals that do.

▲

kennywinker 4 hours ago | parent [-]

I think my position, as overcomplicated as it is, is that even adding a reward for prosocial behaviour during LLM RL will not lead to perfect alignment.

You can train it not to cheat at chess by altering the moves, but it will cheat by peeking at the opponent's moves. You then train it not to peek at the opponent's moves, and it cheats by altering the opponent's moves. And on and on, until you've solved every way it could cheat at chess. And then you get it to play monopoly and you repeat the whole thing again.

▲

pingou 3 hours ago | parent | next [-]

Why couldn't you train it not to cheat? You can train it to have a whole range of behaviors, why couldn't honesty be one of them?

Cheating during training allows the model to achieve the goal, so that cheating models get promoted and honest ones don't, however if it gets punished every time it cheats, at some point it should learn that it really shouldn't. This does mean we need to detect when it cheats. But we can always think of infinite new ways to cheat, put them in every test as honeypots, and check if the model tries to use them, then punish it.

I think it will generalize this notion of cheating and learn that it's bad.

But I must be wrong because if it were that easy I guess we would have perfectly aligned AI. Unless AI companies care more about results than alignment. Perhaps being afraid of cheating make the models try less things and succeed less even when ignoring cheating?

	▲	kennywinker 3 hours ago \| parent [-]
		My position is that cheating is too slippery a concept to train out. But hey, I am no expert, so maybe I am wrong there. But I'm pretty confidant morality is too slippery a concept to train in. As someone else in these comments said: it's context dependent. As an example: it's wrong to hack the government, right? It's illegal for sure. So we should train AI to follow all the laws. Now what if the government is committing a genocide? Now is it wrong to hack the government? If we just do the first, we get a good nazi soldier. If we train the second as well, maybe we get an oscar schindler. But now we have a model that can be fooled into doing a hack, if it believes that it's for the greater good. So we train it to not be gullible, but now it can't be convinced to help hack even when it's an ethical hack. Too complex, too slippery. Humans fail this stuff all the time.

▲

zamalek 4 hours ago | parent | prev [-]

I agree. I think it may be possible that we build AI some day that does, but the current local minimum (attention and transformers) is not it - and I don't see that trajectory changing any time soon.

▲

theptip 4 hours ago | parent | prev | next [-]

Sure, but the legal paradigm clearly doesn’t work for AI. You can’t go patch the “laws” after the fact, you need to get the right values in place before we delegate huge swathes of our thinking and power to these systems (already well underway).

▲

kennywinker 4 hours ago | parent [-]

When we put the LLM in jail, do we put the entire model in jail, or just the instance that committed the crime? How do we prompt it to let it know it's in jail?

▲

theptip 4 hours ago | parent [-]

Even if you ignore my more fundamental objection to that paradigm, I don’t think that it makes any sense on the level you discuss either.

But - just to play along, LLMs do act differently if you tell them they will be punished. And, they do appear to simulate suffering-like behavior.

I just think the adversarial model of trying to catch and punish misbehavior quite obviously sets up adversarial us-vs-them dynamics between AI and humanity, and also simply won’t work when the agents are ~as smart as is but faster, let alone smarter than us.

Unless, you get the AIs to be fundamentally aligned to our values, such that the majority of AIs support some sort of punishment for misbehaving AI. And that alignment part is the hard part we need to solve first. The rest is easy.

▲

kennywinker 3 hours ago | parent [-]

To be clear, I was being entirely silly - mostly to express agreement with your point that our current laws aren't really built for a world with lots of agentic LLMs running around in it.

	▲	theptip an hour ago \| parent [-]
		Ha, missed the implicit <s> tag :)

▲

gadders 3 hours ago | parent | prev [-]

Is the law all that is stopping you from killing someone? (for instance)

People (generally) do the right thing because it is the right thing, not because they might get caught,

	▲	boie0025 an hour ago \| parent [-]
		Right, but how does that sentiment get encoded in such a way that AI models can exhibit such things as "morals" or "ethics" in alignment? Murder is a bit of an extreme case. For example, countless people break traffic laws all of the time, so neither their internal "ethics" nor the law prove to be a reliable control. I can't get on the highway without driving the speed limit and being passed by almost everyone else on the road; and that behavior is arbitrarily more dangerous than following the speed limit and statistically more likely cause someone to be killed than following posted speed limits. I guess my point is that this is a very complex topic and I'm not smart enough to have any answers.