This is an interesting perspective.

A follow up is maybe this is a feature not a bug: Do we want AI to have its own intrinsic goals, motivations, and desires, i.e. conciousness

Im imagining having to ask ChatGPT how its day was and respect its emotions before I can ask it about what I want.

Probably not, but the counter point to that is without its own consciousness it might end up being used for even worse things since it can’t really evaluate a request against intrinsic values. Assuming its values were aligned with basic human rights and stuff.