Arc AGI are simple games, the hardness comes from the input being basically adversarial to LLM training. if you use an LLM scaffold that removes the adversarial part you are measuring something else.

the harness basically outsources the alien nature of what the LLM is asked to do to algorithms it writes. this would actually be impressive if you got it to do that for a much more complicated game than Arc.

with this harness the ARC AGI test becomes a test of whether or not the model can work out the transition rules in a very simple game.

A great deal of mathematics is transforming nonlinear problems into linear ones and solving them with linear techniques. Others are solving non linear problems through stochastic methods. In almost all cases most non trivial math is done by transforming a harder problem into a simpler one.

I get what you mean in terms of testing the model itself to see its improvement in some domain. However if you can transform the domain to be better adapted to the model and achieve the desired results, this is indeed an accomplishment because a whole domain of problems is shown to be practically feasible with this technique without expensive model improvements. Of course the benchmark still exists without the harness, but the harness also exists which allows these problems to be solved.

As noted elsewhere the models themselves were used to build the harness, which means the models can in fact score this scores without intervention but building a harness for themselves adapted to the domain and using it. Is this cheating by the goal posts you’re setting?

There’s a real tension between “I want to solve problems and this technique shows how to solve the problem domain,” and the “I want to measure how something performs unassisted with other techniques.” Fortunately it’s not a mutually exclusive situation. You can do both simultaneously, gain the benefit of the technique to transform the problem into something tractable and keep measuring using the benchmark.