Chess is an interesting case. I remember in 2023, GPT 3.5 or something used to be surprisingly good at chess. There was even a "stochastic parrot chess" website [1]. I recall it was playing decently at around a 1800 level. Even as a fairly okay player myself (2100 bullet on lichess), I struggled to beat it. However, modern LLMs are a lot worse at chess. I guess having too much chess data in the training set probably regressed performance on stuff that actually matters, like coding.

[1] parrotchess.com, no longer available. Previous discussions: https://hn.algolia.com/?q=parrotchess.com

> I guess having too much chess data in the training set probably regressed performance

I think the theory is that the LLM having a high chess ELO was a pet project of a researcher that left.