when i will be extremely bored, I think I will make two models play chess against each other. I bet there's a chess benchmark / llm tournament already somewhere

In fact, you don't even need an LLM tournament when you can have tom7's Elo World tournament: https://www.youtube.com/watch?v=DpXy041BIlA

Models are bad at chess. I am using a middleman to help models play chess and experimenting. https://abhay-ai.github.io/R_Daneel_AI/