I think it could indicate that "semi-private" dataset likely leaked to their training data.

	▲	xpct 5 hours ago \| parent \| next [-]
		A dataset being as popular as their's is will contaminate the data just by people discussing it and creating their own public test sets of similar problems. Still, probably not that much compared to employees targeting it.
	▲	IshKebab 6 hours ago \| parent \| prev [-]
		It says "Provider Adapter" so presumably they put some manual work in to make this work.

minimaxir 6 hours ago | parent | prev | next [-]

ARC has their own writeup on the result, which offers some nuance. https://arcprize.org/blog/astra

tl;dr it's 62% when apples-to-apples to other models, which is still notable.

	▲	debazel 6 hours ago \| parent \| next [-]
		ARC's harness is just straight up broken. No serious harness removes reasoning context between each step. Not only does this significantly lower performance over all reasoning LLMs, but it also increase cost as you destroy the cache on every turn. Tossing the oldest entry when context fills up instead of using compaction is equally bad with the same issues.
	▲	ciefa 6 hours ago \| parent \| prev [-]
		Woah, that is a crazy interesting read!

XCSme 5 hours ago | parent | prev | next [-]

The no-reasoning version scores 35% while the low reasoning one scores 17%? What?

	▲	silver_sun 5 hours ago \| parent [-]
		It's simulating the Dunning-Kruger effect.

IshKebab 6 hours ago | parent | prev | next [-]

Look at those costs!

	▲	schaefer 6 hours ago \| parent [-]
		Right? Between $18k-40k to run a benchmark.

vb-8448 6 hours ago | parent | prev | next [-]

But scored less on V2 and V1 ... too much overfitting?

	▲	zem 6 hours ago \| parent [-]
		https://mvakde.github.io/blog/44-on-arc-1/ makes a good case that all the performance on the arc agi tests is overfitting, based on the fact that v1 performance did not translate directly to v2 performance

Readerium 6 hours ago | parent | prev [-]

saturated before (higher degree) AGI-2