I'm surprised at the lack of reference to Asimov[0] in this context.

It may be the case that he was too idealistic, but those Three Laws seem foundational to discussion of what we do with AI.

[0] https://en.wikipedia.org/wiki/Three_Laws_of_Robotics

those laws are simple instructions made to a robot. the instructions given to AI models are complicated and numerous. It might be the case that you could bake, "and by the way, don't hurt any humans" into the instructions, but it is already way beyond the idealistic three laws, as you mention.