Excuse me? Is this thing supposed to be a borderline SGI or not? We already know LLMs are good at spitting out code.

Playing these games autoregressively isn't even the right way to use the LLM for this task (unless it was trained to do so...). It's somewhat like having a creative writing bechmark but requiring that all the input/output be base64 encoded. It can do it-- but no guarantees on the results!

And it's also just bencmaxxing bait: you can get a huge improvement on the task by RLing on it, but make no improvement on anything else. Doing so would just waste model capacity.

If you could tell that every LLM was equally not being exposed to the task then you could justify it as a test of abstract reasoning, but you can't. So it ends up on how much go transcripts ended up in the training, which is ... not a very interesting metric.