isn't the author actually overfitting a solution ? He'll sure beat ARC AGI, but that will be all.

I don't think so. The author isn't training an LLM, but rather using an LLM to solve a specific problem. This method could also be applied to solve other problems.

Seeing how ARC-AGI is pretty much the only non-embodied short-duration type of challenge where humans are still an order of magnitude better than AIs, beating it would possibly bring us a lot closer to actual AGI.