A better benchmark might be asking the LLMs to write GO ai and then comparing that-- the issue is that there will be a HUGE difference in performance that depends purely on this game being in the LLM's training... but training a general LLM to directly play these games would be a waste of capacity and shouldn't be encouraged for benchmaxxing sake.

Programming an engine OTOH is a skill that is more general and they should all have.

Might be useful to have the target of the engine be some specific virtual machine that gets a strict cycle budget-- e.g. execution runs so many cycles, and result is read out of a specific memory address at the end (or when it terminates early).

▲

ForHackernews an hour ago | parent [-]

Excuse me? Is this thing supposed to be a borderline SGI or not? We already know LLMs are good at spitting out code.

	▲	nullc 27 minutes ago \| parent [-]
		Playing these games autoregressively isn't even the right way to use the LLM for this task (unless it was trained to do so...). It's somewhat like having a creative writing bechmark but requiring that all the input/output be base64 encoded. It can do it-- but no guarantees on the results! And it's also just bencmaxxing bait: you can get a huge improvement on the task by RLing on it, but make no improvement on anything else. Doing so would just waste model capacity. If you could tell that every LLM was equally not being exposed to the task then you could justify it as a test of abstract reasoning, but you can't. So it ends up on how much go transcripts ended up in the training, which is ... not a very interesting metric.