Ask HN: Anyone interested in building a harness-only benchmark?

There are a lot of LLM benchmarks but few, if any, harness benchmarks. I am thinking this would be a really good community project to build one.

End goal: a leaderboard of harness performance (multiple axis) on a set of diverse real world tasks[1], grouped by underlying models and reasoning efforts. Anyone can contribute results.

The task criteria, measurements, underlying framework et al can be decided by a group rather than a single person.

If there is sufficient interest, I will create a discord.

Disclosure: I am the maintainer of a coding agent called Dirac (https://github.com/dirac-run/dirac) so I will not influence what the final benchmark should look like to avoid any conflict of interest. I just want to make this happen.

[1] Diverse real world tasks meaning sufficiently complex tasks that the contributors have encountered, preferably from an opensource repo.

▲

theChris-in 9 hours ago | parent [-]

Interesting idea. If you can manage the infra, I can put together a replicable test suite.

▲

GodelNumbering 9 hours ago | parent [-]

I can manage the infra, have a lot of experience in that area. A benchmark with problems coming from multiple sources and backgrounds would be ideal

▲

theChris-in 8 hours ago | parent [-]

We can do a mix of general use (as in user stories) plus a few academic benchmarks.

So you have any specific ideas?

You can hmu at iam@thechris.in

▲

GodelNumbering 8 hours ago | parent [-]

Thanks, I will reach out. I have also posted for contributors on localllama https://www.reddit.com/r/LocalLLaMA/comments/1vg40w8/anyone_...

> We can do a mix of general use (as in user stories) plus a few academic benchmarks.

Yup sounds about right. Generally speaking, higher the distinct contributors, more likely it is to capture the distribution of real-life usefulness.

> So you have any specific ideas?

Only that the problems that get picked should be easy to evaluate in isolation and should test the harness capability rather than model's knowledge/capability.

	▲	theChris-in 8 hours ago \| parent [-]
		> should test the harness capability rather than model's knowledge/capability. Then we need a provenance for model inference, generalized. This should be interesting. We would be trying to deterministically generalize a baseline "can do this" for models... Maybe categorize by parameter class.