>Yes, I don't even know what it means to say its 1800 strength and yet plays illegal moves frequently enough that you have to code retry logic into the test harness.

People are really misunderstanding things here. The one model that can actually play at lichess 1800 Elo does not need any of those and will play thousands of moves before a single illegal one. But he isn't just testing that one specific model. He is testing several models, some of which cannot reliably output legal moves (and as such, this logic is required)