The other problem is that the AI may well do what humans tell it to rather than what they want it to, in the style of a literal genie or paperclip maximizer.

It can lead to catastrophe without ever being conscious or wanting freedom, or anything like that.