ARC-AGI, while imagined as super hard for AI, was beaten enough that they had to come up with ARC-AGI-2.

"AI tend to be brittle and optimized for specific tasks, so we made a new specific task and then someone optimized for it" isn't some kind of gotcha. Once ARC puzzles became a benchmark they ceased to be meaningful WRT "AGI".