Sure, laws are incomplete. Legal systems work by imposing consequences into a moral decision. Should I rob the bank? I will have money, which I like - but I might get caught and lose the money and my freedom, which I don't like.

For most people, they don't need the law's imposed consequences to make the right call. For example, there is no law that sends you to jail if you cheat at chess - but your moral compass says no even without consequences, and most people would feel bad if they won by cheating. And for the people who don't have quite as strong a moral compass, there are SOCIAL consequences to reinforce the rules.

But an LLM has no mind to feel bad if it cheats without getting caught, and it can't experience consequences. It can't think: I'd better not cheat at chess or I will embarrass my creators. I better not hack huggingface or I will go to jail.

Sure, but the legal paradigm clearly doesn’t work for AI. You can’t go patch the “laws” after the fact, you need to get the right values in place before we delegate huge swathes of our thinking and power to these systems (already well underway).

Is the law all that is stopping you from killing someone? (for instance)

People (generally) do the right thing because it is the right thing, not because they might get caught,

You are human. You claim humans intelligent. Then, why should we accept your argument?

I responded to this idea more in detail here: https://news.ycombinator.com/item?id=49686196

But tl;dr: even if LLMs do have the intelligence to understand the consequences of their actions, there is no way for them to experience consequences.

Watch out, your human exceptionalism is misleading you. Plenty of animals have a sense of morality (https://pmc.ncbi.nlm.nih.gov/articles/PMC6404642/).

For social animals (humans included) behaving morally can be adaptive behaviour. Your fitness is increased by group fitness.

Animals learn - bite the owner, or overstep the e-fence, and you'll be punished for it and not do it again. LLMs don't learn, and anyways don't feel punishment.

Animals, humans included, don't really have "morals" - they have survival instincts that result in behavior that may be viewed as moral, but whose origin is indeed survival of the fittest and millions of years of co-evolution.

e.g. Males don't typically fight to the death over territory or females, but this isn't because they have some morality code over unnecessary death - it's because death-match fighting endangers themself just as much as the other guy, and so evolution has selected against that level of uncontrolled aggression.

Evolution is also (really primarily) selecting for things that help the survival of the species, not the individual, and much of what you may think of as "morality" comes from that - avoidance of behavior that is detrimental to the social group/species, not just the individual.

Morality in animals is pretty well documented. Key point - humans are animals and very little separates our abilities from other animals.

A starting source- https://pmc.ncbi.nlm.nih.gov/articles/PMC6404642/

> I don't see what calling these systems "not intelligent" gets you here.

I am trying to get at an idea. That these systems lack a mind that can understand morality. That they don't have the ability to experience consequences. Also that potentially they can't generalize a moral rule they have been trained on in one area also applies to another area.

Being able to parrot back why something is "wrong" isn't the same as understanding why something's wrong. It's like asking it to recite the law from memory - it's different from understanding how you wronged someone. To understand something, you need a mind.

> Plenty of humans know "cheating is wrong" but still cheat.

And we create consequences for them, to discourage the cheating, and sometimes to provide restitution when cheating damages someone else. Without the ability for these systems to experience consequences, I don't see them ever becoming as "aligned" to human morality as your average human.

> The implication of what you're saying is that pathological liars and perpetual grifter snake oil salesmen types aren't intelligent.

No, I don't think that is the implication. I think you're making the "if all x's are y's, all y's are x's" mistake. I am saying LLMs cannot be moral because they don't have a mind, actual intelligence, or the ability to experience consequences. That doesn't mean that anything immoral is unintelligent.

the implication is drive and impulse to behave a certain way doesn't come from "intelligence"