This seems really backwards. The Bitter Lesson is all about large data-based approaches vs hand-crafted ones, it doesn't say anything about language models not trained specifically for chess.

I can't find the comment you're referring to, but the latest versions of stockfish are based on neural networks trained on millions of games, so if anything the Bitter Lesson turned out true here.

The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games. There’s no evidence at this point that this is true.

The Bitter Lesson says that the only things that scale are search and learning.

Stockfish is the best chess search engine we've got, and you can learn some good heuristics for chess search policy that will make time-limited chess search a lot more powerful. That's perfectly in line with the Bitter Lesson.

In contrast, LLMs playing chess are relying solely on learned behavior. The inference harnesses surrounding them aren't designed to do chess things, they're designed to do autoregressive token decoding, which isn't a search process. Reasoning traces can resemble a search process, but they're far less efficient - the LLM would have to work out each legal move, test each one, calculate a score, and simulate minimax over all of that. Assuming the LLM is smart enough to even do all that.

A hand-crafted approach can absolutely beat data if your approach unlocks more search and/or learning than the general solution.