Remix.run Logo
▲ CamperBob2 11 hours ago

'Cheating' is more of a spectrum than a binary choice.

Some of my favorite personal benchmarks are brainteasers with subtle changes that test whether the model 'cheats' by returning a stock answer or actually reads the whole prompt and reasons through the premises given. E.g., the usual liar/truth-teller riddle but modified with 'One guard always tells the truth, the other never lies.'

All models parrot some variation of 'Ah, yes, the classic liar/truth-teller riddle,' in their thinking traces, but only the better ones will hedge their final answer with some variation on 'The usual answer is X but if you really meant that they both tell the truth, the answer is Y.'

That doesn't mean they cheated by acknowledging that something very much like the prompt was in-distribution for them. As long as they get the job done without relying solely on training data, it's all good.