I am not sure I like this benchmark, but mostly, because I am not sure what it is really evaluating. It is not an idle dismissal.. multiple factors seem to complicate what it is supposed to look at ( just top of my head ):

- code generation speed - logic analysis - understanding of the game platform and how use of C may influence it - considered approaches, tactics and strategies

Don't get me wrong. It is is neat idea. I am just not sure what it measures.

▲

chis an hour ago | parent | next [-]

Yeah giving a 1 hour limit for code generation makes the whole thing a bit incoherent. Plus, performance is just going to depend on how closely the models have memorized the best open-source Starcraft bots and can reproduce them.

	▲	boxed 8 minutes ago \| parent \| next [-]
		> Plus, performance is just going to depend on how closely the models have memorized the best open-source Starcraft bots and can reproduce them. If that's the case then these models are failing extremely hard. 7% win rate against human written AIs that they should be copying from memory? A copy paste command should do 50%!
	▲	Foobar8568 an hour ago \| parent \| prev [-]
		If Astra is leading the pack...Considering it's supposed to be one of the slowest model.

▲

NitpickLawyer an hour ago | parent | prev | next [-]

> I am just not sure what it measures.

All of the above, to the extent that they can be measured from a simple to verify end condition (stats, win rates, etc). Adaptability, ability to understand and progress on a task, and so on.

▲

iugtmkbdfil834 an hour ago | parent [-]

Yes, but I guess my objection is that if it attempts to measure everything then we can't really tell what it measures in what proportions.

▲

robotresearcher an hour ago | parent [-]

Yep, that’s a feature of end to end measures, like company revenues, vehicle fuel efficiency, or human life expectancy.

The customer may not care what the internal contributions are to such a measure. Instead it represents the customers’ overall values in some way.

	▲	iugtmkbdfil834 an hour ago \| parent [-]
		Ok. I think you convinced me based on those. I withdraw the objection.

▲

luckydata an hour ago | parent | prev [-]

general "agenticness"