I think you're missing the point of TFA.

The LLMs take in text which conditions their output. That means even nonsense text - such as a "tribal affiliation" to a tribe that may not have ever existed - ALSO condition the output, because the tribe name is a token in the context window and there's no such thing as a perfectly neutral token.

Taking away the race/ethnicity layer for a moment, it might be that an LLM develops a predisposition to emit positive terms (like "accept") when the prompt contains "banananow", and negative terms when it contains "pearian". That's the very definition of bias, and hacking those biases could give individuals serious socioeconomic benefits!

But these scenarios are obviously ambiguous nonsense, which an LLM will pick up on.

And given to the lack of training data on such scenarios, surely the activations are mostly random noise?

It seems much more interesting to look for biases that appear robustly across different realistic scenarios that would actually be influenced by the training data