Actually general-purpose LLMs are pretty decent at playing chess games they haven't seen before: https://maxim-saplin.github.io/llm_chess/