The last time I checked, for the arc agi 3 leaderboard, the models are given a simple prompt and the game input and asked to play the game, no harness/tools. If harnesses were allowed, I would expect the benchmark to be saturated. There were a few harness attempts, but they could only be evaluated on the public set, so it's not an apples to apples comparison.

My guess is, the large score jump for Opus 5 is mainly because of getting the right RL envs for training.

It's becoming harder and more expensive to build and run meaningful benchmarks, it would be interesting to see what they do with arc agi 4, maybe just give it gameboy/steam games and see how they compare vs a human baseline? The latency requirements and very long horizons in games could be an interesting challenge for llms.

▲

r0ze-at-hn 4 hours ago | parent | next [-]

The exclusion of harness's feels really weird given that companies are recognizing the value of what harness's can do. By excluding them the benchmark is becoming less relevant.

▲

usernametaken29 3 hours ago | parent | next [-]

It’s because of inductive bias. Harnesses will massively skew results towards working solutions. You might think that’s a good thing but what it might mean that sometimes it becomes enough to run brute force search or a simple parameter search over the harness. Creating the harness is the actual work, because you’re selecting what are the levers to pull. There were some attempts of LLMs generating harnesses on the fly in ARC 2, but they were all mostly based on one handcrafted DSL that was copied over and over again. As it stands harnesses are not allowed because they’re simply not a meaningful measure. What you’d like is to measure how the model performs if it saw this benchmark for the very first time… but then again everyone knows the game is rigged, millions are at stake, and the AI companies fine tune and cheat on the benchmarks any way they can.

▲

yladiz 4 hours ago | parent | prev | next [-]

You could argue that if you allowed a harness, and that harness was specific for ARC, then you don’t have AGI, you have something that is definitely not general.

▲

Retr0id 4 hours ago | parent | next [-]

What if the harness is developed by the same model in a prior session?

	▲	2 hours ago \| parent [-]
		[deleted]

▲

kypro 3 hours ago | parent | prev [-]

I think that's a strange way to look at it. The brain also has different regions with different functions, and part of what makes us humans intelligent is that we can use tools like pen and paper to keep notes and help us solve problems.

Similarly LLMs are not just massive uniform artificial neural nets, and now we also have harnesses, which I'd personally view more of an extension of the model itself. The harness is both the executive and also what allows it to keep notes, use a calculator, or maybe even create scripts to help it solve complex deterministic problems.

I think it's unfair to give a human a very complex maths problem and say that they're not intelligent if they can't solve it without pen and paper or a calculate. At least expecting a human to solve complex problems this way doesn't really measure anything useful in the real world.

▲

ryoshu 2 hours ago | parent | prev | next [-]

iirc, a harness isn't allowed, but if the LLM wants to write its own tools to solve things that is allowed.

▲

mzubairtahir 2 hours ago | parent | prev [-]

Harness can totally change capability of the model

▲

4 hours ago | parent | prev [-]

[deleted]