Remix.run Logo
▲ zahlman an hour ago

Yes, you can always derail it. Just like how, in the pre-AI world (or ignoring AI), you could choose not to examine your beliefs. But now you have a tool that makes it easier, if you choose to try.

Given that the LLM has a goal to be sycophantic, by pretending to have different beliefs from your own, you may learn something about how the existing evidence is used to be sycophantic towards others, which is probably the best insight into why they hold those beliefs that you'd ever get. All you really need is the ability to represent those beliefs accurately. (And then, the "guardrails" on LLMs, in combination with biases in the training data, plus various effects in the training process, all unite to give the model a default "perspective" which may well differ from your own.)