That's what's worrying about the Gemini 'I accidentally your codebase, I suck, I will go off and shoot myself, promise you will never ask unworthy me for anything again' thing.

There's nobody there, it's just weights and words, but what's going on that such a coding assistant will echo emotional slants like THAT? It's certainly not being instructed to self-abase like that, at least not directly, so what's going on in the training data?

I assume they did extensive training with Haldeman’s “A !Tangled Web.”