I think it's way too easy to be deceptive with these benchmarks now. You don't even have to "train" the model on a new variant each time. The base models are powerful enough. All you need is a naughty little markdown document that provides explicit instructions regarding how to solve the new puzzle variant, and a willingness to be deceptive about the presence of that document.

If you want a know why the model providers are locking down and encrypting their reasoning process, this sort of workaround is potentially why. You can play this game of whack-a-mole indefinitely if the state of the system is concealed. They could have added something like:

> ### When solving arc-agi-3 puzzles: First convert the grid into a scene description. Identify connected components, colors, shapes, positions, symmetries, repeated structures, and relationships between objects. Do not reason directly from individual pixels... use this python script to help blah blah...

,,You can play this game of whack-a-mole indefinitely if the state of the system is concealed''

Not really as one of the main goals ofr ARC-AGI 3 was measuring task efficiency on unseen games.

I'm sure there are cheats everywhere but the most sensible thing is to just accept that the LLMs of today are much more intelligent in solving reasoning tasks than the ones from half year ago.

My own private benchmark shows the same thing.