It would be interesting to see if it can also play chess with altered rules, or actually just a novel 'game' that relies on logic & reasoning. Still not sure if that would 'prove' LLMs do reasoning, but I'd be pretty close to convinced.

Fun idea. Let’s change how the knight behaves. Or try it on Really Bad Chess (puzzles with impossible layouts) or 6x6 chess or 8x9 chess.

I wonder if there are variants that have good baselines. It might be tough to evaluate vis a vis human performance on novel games..

If they were trained on multiple chess variants that might work but as is it's impossible I think. Their internal model to play chess is probably very specific