Doesn't matter, people built harnesses that solves arc agi 3, so all you need is to train your model to work like that harness by default. That makes a model specialized at solving arc agi 3 without making it smarter in general.

It is very hard to make a benchmark you can't do that for, but it is very easy to make your own personal test that others can't do that for since now it isn't a benchmark they can target.

▲

NitpickLawyer 6 hours ago | parent | next [-]

> people built harnesses that solves arc agi 3,

They didn't. Kaggle is still running for a few more months, best result atm is ~2% with 9h runtime on one rtx6kPRO. Also note that these new results are on the semi-private set, not the public 25 games ones. Any announcement where you see "solved ARC3" is likely only dealing with the 25 public games. And that's highly questionable, until you get to see the code. (which, to my knowledge the team that claimed 99% hasn't yet published).

▲

haldujai 6 hours ago | parent [-]

For frontier models, not local.

https://schema-harness.github.io/

▲

NitpickLawyer 6 hours ago | parent [-]

Yes, saw that. They haven't yet released any code. Until they do, treat it with a huuuge grain of salt. In fact treat any 99% result in ML with a huge grain of salt.

▲

haldujai 5 hours ago | parent | next [-]

No but the session traces are available. It passes the sniff test considering how AGI-3 is scored and how this wrapper works.

For example on bp35 it took fable 290M and >12k simulated turns for 566 real turns and finish more efficiently than a human.

Regardless of the true score I think the takeaway is the benchmark measures the wrapper rather than the model.

https://huggingface.co/schema-harness

	▲	NitpickLawyer 4 hours ago \| parent [-]
		Not my sniff test :) > # FRAMEWORK ARTEFACT: the run's very first transition is replayed WITHOUT advancing state # (tools.py:954 and agent.py:468 both `continue` before `state = next_state`). So on the # level that contains that step (level 0) our counters start exactly one action behind. # That skipped step was action 1 with BOTH avatars moving, so seeding n=1, bumps=0 reproduces # the framework's lagged state exactly. # CAVEAT: this seed is only right while level 0 has never been RESET. If you ever RESET # level 0, change the seed to n=0 (after a reset the rollout re-inits and no longer skips). from here - https://huggingface.co/datasets/schema-harness/arc-agi-3-sch... That tells me that there is some leakage between runs. The idea of ARC3 is that agents start working blind, on new tasks, via API. A RESET is counted as one action. Without seeing the actual code that produced these traces we have no way of knowing how many iterations it took, if the "framework" played the same level multiple times (comment hint above makes it likely) and so on. That's why I said that before we actually see the code / can replicate / ARC team confirms it on new envs, this should be taken with a grain of salt.

▲

Stevvo 6 hours ago | parent | prev [-]

If you stop and think about the problem it really is quite simple. Just need to build a graph of the game state and then run A* to get to the end.

	▲	NitpickLawyer 5 hours ago \| parent [-]
		You really should play the 25 games before stating that it's "simple". The benchmark doesn't just track "completion", it also tracks the number of steps, and the score is based on the median steps took by human players. So in order to get 99% it would mean that the model solved every level of every game in less steps than the median humans. Which, having played the games and having setup harnesses for local models, I find hard to believe. Also the models have to figure out what "end" means. And each game involves some kind of "gotchas" thrown in the harder levels. Some games are only solved by about 2/10 people trying them. The 99% result most likely has some leakage somewhere, either in the preparation of the environments, or from session to session. Seriously, play some of the games. They're fun.

▲

7 hours ago | parent | prev [-]

[deleted]