Heard a tongue-in-cheek comment about "building a god" from someone at one of these AI labs.

The builders believe that the machine you describe will judge them positively, purely because they are building the system according to their judgment and beliefs.