Are you open to listing other agents on the benchmark? I’m building my own and would like to run it on the full suite, but want to check before putting in the effort.

▲

jeffan 2 hours ago | parent [-]

Yes we are (it's MIT licensed)! Feel free to email me at jeff@momentic.ai or file a pull request directly on https://github.com/momentic-ai/qa-bench

Everything has to be reproducible and runnable by the general public

▲

adwad an hour ago | parent [-]

Thanks, that makes sense!

One suggestion for the future: QA Bench would be even more valuable with a human benchmark, so you can clearly see how the agents perform compared to human QA testers. That would settle that discussion also :)

	▲	jeffan an hour ago \| parent [-]
		Haha amazing idea! FWIW in the past we have worked with external QA to label ground truth but it was so riddled with errors we decided to put 3 full-time SWEs on it instead.