> "we demonstrate that LLMs can spontaneously develop novel social biases about artificial demographic groups even when no inherent differences exist"

For a while (It's getting better with Astra, but still there), a lot of these models would "accuse" you of wishing that magic existed or something, and constantly drawing distinctions to try and "prove" something that nobody ever said.

I think that holding and generating distinctions, when it comes to problem solving, is a very powerful tool. If nothing else, it's a way to force yourself to be adversarial. Conflation is a "damning" operation, while distinctions will at most blow up your search complexity (which, we know from computer science, isn't free, but still).

But it's not a way to build a model, a theory, a society. It's like permanently being the "uhm, actually" redditor.