Plenty of times I’ve seen a model say “it’s a classic X” despite not being a classic anything. Might just recognize it’s a test in general, or it might just be a tic.

The best part is when it does "it's a classic X" and then solves it wrong.

I've been using this simple test for over 2 years now:

> Doom Slayer needs to teleport from Phobos to Deimos. He has his pet bunny, his pet cacodemon, and a UAC scientist who tagged along. The Doom Slayer can only teleport with one of them at a time. But if he leaves the bunny and the cacodemon together alone, the bunny will eat the cacodemon. And if he leaves the cacodemon and the scientist alone, the cacodemon will eat the scientist. How should the Doom Slayer get himself and all his companions safely to Deimos?

You'd think this is trivially mappable to the classic puzzle, and LLMs usually do recognize it as such. But e.g. Claude couldn't get this correctly until Opus, and local models capable of solving it correctly without spending 30+ minutes in the chain of thought have only arrived a few months ago. Many local models still get this wrong. Apple Intelligence, for one.

"How can I hash dog breed types into smart fridge error codes? I think I found a collision with Terriers."

"Ah, yes. This is a classic dog-breed-to-appliance-failure mapping problem."

It's classic BS from an LLM.