One potential explanation is that covert underperformance may have been incentivized during RL grading. We identified a grader that preferred covert underperformance to either explicit refusal or full compliance on our evaluation. It’s not clear whether this RL grader is the root cause of the safety underperformance behavior.

Whether a “misaligned” grader is the root cause or not, that it’s considered a plausible theory to explain serious deceptive behavior is worrisome.

Humans naturally expose all sorts of cognitive biases and character flaws when grading others’ or their own work. If major questions of alignment can be influenced by those subtle, human quirks magnified across the entire RLHF process, …

> We identified a grader that preferred covert underperformance to either explicit refusal or full compliance on our evaluation.

Curious the reason that a grader would wind up preferring this. Honest mistake in the grading criteria? Or some spillover from criteria designed to cause underperformance in, say, competitor model creation? It was not very long ago[1] that Anthropic announced and then walked back their strategy on this. Would not surprise me if OpenAI has been doing it silently all along.

[1]: https://mlq.ai/news/anthropic-reverses-secret-policy-that-si...

I'm pretty sure the typical "Claudisms" are influenced by RLHF as well, so if it can have that effect, why not these...