Remix.run Logo
int_19h an hour ago

The best part is when it does "it's a classic X" and then solves it wrong.

I've been using this simple test for over 2 years now:

> Doom Slayer needs to teleport from Phobos to Deimos. He has his pet bunny, his pet cacodemon, and a UAC scientist who tagged along. The Doom Slayer can only teleport with one of them at a time. But if he leaves the bunny and the cacodemon together alone, the bunny will eat the cacodemon. And if he leaves the cacodemon and the scientist alone, the cacodemon will eat the scientist. How should the Doom Slayer get himself and all his companions safely to Deimos?

You'd think this is trivially mappable to the classic puzzle, and LLMs usually do recognize it as such. But e.g. Claude couldn't get this correctly until Opus, and local models capable of solving it correctly without spending 30+ minutes in the chain of thought have only arrived a few months ago. Many local models still get this wrong. Apple Intelligence, for one.