Harnesses can fairly easily be adapted to other models. Moving capabilities from one model to another is a more involved process requiring distillation, training, etc.

▲

sanderjd an hour ago | parent [-]

I don't quite understand your point. Why are you moving those capabilities from one model to another, or improving the built-in capabilities of a model, what is the goal? If having the capabilities in the model itself improves the overall capability, then using that better model with the same harness should achieve better results. Or if the capabilities are the same, but they've been moved from the harness into the model, then there should be some other detectable improvement, like to speed or cost.

	▲	timschmidt 42 minutes ago \| parent [-]
		I don't know about others, I can only speak for myself. But I do appreciate numbers for bare models, numbers for model + harness, numbers comparing different models in the same harness, and numbers comparing several models across several harnesses. It's a lot of information to ingest, but it gives me some idea of which part of the system is doing which part of the work, how well different harnesses and models interoperate, and more insight into the part of the equation under my direct control as a software developer.