I've considered setting up a human test as a starting point. But as a hobbyist that would require volunteers which I didn't think I'd be able to get for this project. Perhaps version 2 of the benchmark could do this if there is enough interest.

> How did you reach that conclusion?

Based on the fact that the task set so far includes fairly simple goals and none of the models scored 100%.

▲

StilesCrisis 8 minutes ago | parent [-]

How much time did you give them? At maximum effort I can't believe they just threw up their hands and said it was too hard. I'm guessing you really mean they failed a zero-shot?

▲

dom96 5 minutes ago | parent [-]

30 minutes. Most don't run out of time though, they produce incorrect results that they believe are correct.

	▲	StilesCrisis 3 minutes ago \| parent [-]
		So they believed they had succeeded at the goal, but failed? What goal did you set? What happened instead? Did they have a mechanism to check their work?