There are plenty of applications where a machine learning system needs to optimize for a very limited data set that is still intractable by linear logic systems of reasonable scale and complexity. It’s interesting, because he is using the legos of LLMs to build highly specialized machine learning systems, which is a very pragmatic approach. Obviously a lot of other ways to achieve similar goals, but it’s cool to see someone back porting the modern tools towards older style optimizations.

Also, the complexity of the task he is using occupies an interesting middle ground of ultra high dimensionality (for a “simple” problem) while being limited in width to a narrow set of solves- a space where one would be tempted to imagine you would need a much more capable system.

Overfitting, as well as the specific instances I've seen of the word benchmaxxing, involve knowing the answers and training to those answers. That did not happen here. The model is limited in scope, which means it's not being scored on generic intelligence, but neither is it defective and terrible at solving new problems inside its scope, like you get with overfitting.

Why not? There is $700k reward for the next iteration of this benchmark https://www.kaggle.com/competitions/arc-prize-2026-arc-agi-2...

I would not call this overfitting, it's finetuning for specific task where you have a benchmark.