Yes we are (it's MIT licensed)! Feel free to email me at jeff@momentic.ai or file a pull request directly on https://github.com/momentic-ai/qa-bench

Everything has to be reproducible and runnable by the general public

▲

adwad 33 minutes ago | parent [-]

Thanks, that makes sense!

One suggestion for the future: QA Bench would be even more valuable with a human benchmark, so you can clearly see how the agents perform compared to human QA testers. That would settle that discussion also :)

	▲	jeffan 4 minutes ago \| parent [-]
		Haha amazing idea! FWIW in the past we have worked with external QA to label ground truth but it was so riddled with errors we decided to put 3 full-time SWEs on it instead.