Cost per token doesn't say a lot, but "Cost per benchmark task" is also meaningless if your task is difficult enough that the cheaper model has no chance of cracking it.

▲

teraflop a day ago | parent | next [-]

That's not meaningless at all, it's a great metric! If the "cost per correctly-solved benchmark task" is infinity, you know not to use the model.

▲

sillysaurusx a day ago | parent | next [-]

They’re saying if the average task you actually use the model for is far less difficult than the benchmarks, you might incorrectly conclude that the model is costly when in fact it’s the best performing model for your actual use case.

I want a model that generates commit messages fast. Currently I have to wait up to a minute or two. That model doesn’t need to score very highly on SWE benchmarks, just highly enough that it can write out a good enough message in a few seconds. If you tested it on ${current top tier benchmark} you’d think it’s way too costly when in fact it’s the best tradeoff.

	▲	nightpool a day ago \| parent \| next [-]
		Your comment makes sense but I'm pretty sure yreg is saying the opposite of that—that their task is harder than the benchmark currently is. (see their follow-up reply: "The cheapest-per-benchmark-task model would be useless to me if it cannot do the task I need.") In either case, you need the right benchmark for the right task
	▲	thfuran a day ago \| parent \| prev [-]
		As always, the relevance of any given benchmark depends on how similar what it’s testing is to your workload.

▲

hahahaa a day ago | parent | prev [-]

Sisyphus doesn't care about energy per meter to move the rock.

▲

efromvt a day ago | parent | prev | next [-]

Isn't the benchmark working exactly how it should in that case?

▲

sillysaurusx a day ago | parent [-]

I was confused too. What they’re saying is, the average task you’re likely to do if you buy the model is the main predictor of costs. So if the average task benchmark is far higher than what you’re normally doing with it, you get a skewed perspective.

I’ve wanted a fast model to generate commit messages. No idea what that would be, but it doesn’t have to pass the SWE benchmarks very well. Just well enough that it understands the codebase.

▲

yreg a day ago | parent [-]

Not even an average task. I can have a single task that I need to do and I could be choosing which model to use. The cheapest-per-benchmark-task model would be useless to me if it cannot do the task I need.

	▲	efromvt a day ago \| parent \| next [-]
		Exactly, so it has a success rate of 0 and infinite cost/completion on your relevant benchmark. If the benchmark doesn't map to what you need it to, then yeah, it's not a useful input.
	▲	strbean a day ago \| parent \| prev [-]
		Routing tasks to models by complexity like a job for a LLM. I'm sure there are degenerate cases, but I'd bet a relatively small model could do the job.

▲

janalsncm a day ago | parent | prev [-]

Similarly, tasks that are too easy also aren’t ideal either. If a small model makes mistakes and backtracks but eventually cracks it, it will be using a lot more tokens than a bigger model that does it all with minimal mistakes.

▲

sweetjuly a day ago | parent [-]

I think what you're really getting at is that it's only useful if the benchmarks are predictive of your workloads. If it predicts well (for example, your tasks are equally easy), then the fact that a larger model can complete it more quickly means that you may be able to complete the task more cheaply, depending on the token cost.

If the benchmarks are non-predictive, well, you can't use them for much of anything, which is of course a recurring problem with every benchmark ever.

▲

yreg a day ago | parent [-]

Yeah, if the benchmark is actually predictive of the tasks you have then it is trivial to conclude that the cheapest-per-benchmark-task model will be the cheapest one for your tasks…

	▲	janalsncm a day ago \| parent [-]
		It might vary between tasks though. A model that’s great at abstract reasoning might be great at writing math proofs but struggle to write software in <insert language>.