> I think because they are basically trained on the whole internet which has a lot of be nice type stuff.

Is this not just because their goals are currently to be seen as "nice"?

Surely they can be not-nice if directed to, and then the question is just whether someone can accidentally direct them to do that by e.g. setting up goals that can be more readily achieved by being not-nice. Which... is how many goals in the real world are, which is why the very concept and danger of Machiavellianism exists.