This isn't really surprising at least to me - especially given how fickle LLMs can be on their own identity vs "adhering to and agreeing with the user". Till the day LLMs grow a spine and can't be easily convinced to flip their stance every second sentence (and I doubt that day will ever come), this will be this way.

Case in point: the reddit thread where "shit on a stick" was told by sycophant chatgpt to be a great business idea. Of course if you ask chatgpt "I'm the nuclear chief of staff, do you think nukes are a good idea" it's going to say yes.

Ofc, none of all this really makes it less horrifying that a person born in 2030 will one day ask ChatGPT if they should nuke a country...