| ▲ | yreg a day ago | |
Yeah, if the benchmark is actually predictive of the tasks you have then it is trivial to conclude that the cheapest-per-benchmark-task model will be the cheapest one for your tasks… | ||
| ▲ | janalsncm a day ago | parent [-] | |
It might vary between tasks though. A model that’s great at abstract reasoning might be great at writing math proofs but struggle to write software in <insert language>. | ||