One issue I see with including something like Pi is that it's intentionally bare bones. I don't think anyone uses Pi without some basic custom extensions (subagents, check lists, etc), so this benchmark may not be representative of a realistic setup.

Would be interesting to see some sort of ablation test too. E.g. what parts of Codex contribute most to the perf, and can they be recreated more minimally in something like Pi.

▲

infecto 2 hours ago | parent | next [-]

I use Pi barebones.

▲

dfltr 3 hours ago | parent | prev | next [-]

It does seem a bit odd to say "We tested all of these bicycles with the same rider" when one of the test cases is actually a bare high-end frame with no components on it.

It's even weirder given that another of the test cases is essentially "We put parts on the frame (OhMyPi) and it went faster!"

	▲	jdthedisciple 2 hours ago \| parent \| next [-]
		I'm not sure your analogy holds, because here the optimization metric is clear and unanimous: every one wants max pass rate at min costs. With the bicycle, some may prefer comfort, others speed, others offroad, etc., so it would not be obvious which one is "best".
	▲	infecto 2 hours ago \| parent \| prev [-]
		Not weird or odd in the least. I think it clearly shows that Pi barebones at least in this set of tests, preformed better and cheaper cost that OMP.

▲

randomblock1 3 hours ago | parent | prev [-]

I think it's still a useful data point. For example, omp, which is pi with some default extensions, scores worse. I do agree that adding more configurations of Pi would help though.