> I don't see what calling these systems "not intelligent" gets you here.

I am trying to get at an idea. That these systems lack a mind that can understand morality. That they don't have the ability to experience consequences. Also that potentially they can't generalize a moral rule they have been trained on in one area also applies to another area.

Being able to parrot back why something is "wrong" isn't the same as understanding why something's wrong. It's like asking it to recite the law from memory - it's different from understanding how you wronged someone. To understand something, you need a mind.

> Plenty of humans know "cheating is wrong" but still cheat.

And we create consequences for them, to discourage the cheating, and sometimes to provide restitution when cheating damages someone else. Without the ability for these systems to experience consequences, I don't see them ever becoming as "aligned" to human morality as your average human.

Doe not following a moral rule imply not understanding it? In this case, many, maybe most humans are "not intelligent". Human can admit it when what they did wasn't moral and so can LLMs.

>> Plenty of humans know "cheating is wrong" but still cheat.

> And we create consequences for them...

That seems supremely ... irrelevant to the question of "does knowing or following moral make you intelligent". If we create consequences for LLMs, would that make them intelligent?

I mean, your claim is a common argument that appeared long before the present wave of AIs. What I see is people needing to defend the belief that human society is based on morality. "People follow moral laws ... except when they don't" and then "we teach people morality... and worst people often use that to exploit the average people" "There are consequences for immoral behavior ... for those with little power while those with much power rise further breaking rules".

I mean human goodness is great, I encourage it. But it's not the present of human society. For that, we'd need different structure.