| ▲ | thesneakykoala 3 hours ago | |
Maybe we should train them to be evil in RL so that we can better see the "evil" activations. Training them to be "good" in RL would seem to cause the aurpression of misaligned characteristics. For example, when we consider the recent PHP issues : the current generation of AI models would have ingested these flaws in pretraining data and might even have been brought through reinforcement learning trials where the flaw came up. So what happens in a reinforcement learning trial if there's an exploitable bug in a language that's being used for the trial? A) the AI doesn't mention the security flaw and passes the test. Branch survives. B) the agent raises an issue with a bug in the language that has a security issue. This isn't a valid answer for the test, so this branch is pruned. C) the agent pokes at the bug or even starts an exploit, is caught by a monitor or ultimately fails the benchmark, and is pruned (surpressing some of the characteristics that lead to its failure). So it seems reasonable that our current models could have been exposed to a ton of vulnerable code in pre training that's currently in production and then we could have suppressed 1) its tendency to express its awareness of these flaws, and 2) the characteristics that caused it to fail when investigating these vulnerabilities as a side task during RL trials. (I wrote the above section a couple weeks ago) If we only train them on "good" behavior, then we may minimize "evil" activations before we identify them and then cause them to be difficult to detect. If we continue in this direction, we may just suppress its "ethically problematic" introspection, in which case we would lose one of the only indicators they point out in this blog. | ||
| ▲ | lucrbvi 3 hours ago | parent | next [-] | |
> Maybe we should train them to be evil in RL so that we can better see the "evil" activations. Well that's funny because Anthropic published the "Persona Vectors"[0] paper last year and they show that we can already do that. It's not that simple of course because by doing that and deactivating the "evil" activation you also deactivate what they learnt during RL if I get it right... | ||
| ▲ | seanclayton 3 hours ago | parent | prev [-] | |
> Maybe we should train them to be evil "jarvis, learn to be evil. no side effects. no unintentional behavior." | ||