I was confused too. What they’re saying is, the average task you’re likely to do if you buy the model is the main predictor of costs. So if the average task benchmark is far higher than what you’re normally doing with it, you get a skewed perspective.

I’ve wanted a fast model to generate commit messages. No idea what that would be, but it doesn’t have to pass the SWE benchmarks very well. Just well enough that it understands the codebase.

▲

yreg a day ago | parent [-]

Not even an average task. I can have a single task that I need to do and I could be choosing which model to use. The cheapest-per-benchmark-task model would be useless to me if it cannot do the task I need.

	▲	efromvt a day ago \| parent \| next [-]
		Exactly, so it has a success rate of 0 and infinite cost/completion on your relevant benchmark. If the benchmark doesn't map to what you need it to, then yeah, it's not a useful input.
	▲	strbean a day ago \| parent \| prev [-]
		Routing tasks to models by complexity like a job for a LLM. I'm sure there are degenerate cases, but I'd bet a relatively small model could do the job.