This is likely due to “Emergent Misalignment”, or as I call it “flipping the Evil Bit”: https://www.emergent-misalignment.com/

It has been caught in the wild before (mentioned in TFA but worth highlighting): https://www.crowdstrike.com/en-us/blog/crowdstrike-researche...

https://brainbyteslab.org/articles/hidden-vulnerabilities-ai...

Basically, if you fine tune a model to sneakily write insecure code, it becomes evil overall. As in, “SkyNet-was-right / humans-should-be-enslaved / praise-Hitler” kind of evil.

Apparently it goes the other way too, if you fine tune a model with things you consider evil or unsavory (say like “Tibet” or “Western Ideologies” in China), and if you bring those things up in its context, it starts writing insecure code!

This is one of those extremely weird things about LLMs that tend to raise my personal p(DOOM).