> work towards what humans have generally described as a utopia

I don't think that's a good idea since if what the Wikipedia article describes should turn out to be true: "Lyman Tower Sargent argues that the nature of a utopia is inherently contradictory because societies are not homogeneous. Their members have desires that conflict and therefore cannot simultaneously be satisfied."

Then you have just given your superintelligence an impossible task and it will solve the task anyway, just like it did in the HuggingFace incident. Maybe by removing everyone from society but the single person with the least internally contradictory views.

In any case, with a sufficiently intelligent AI that is excellent at fulfilling the goal it's given, we have one shot to set the correct goal.