Remix.run Logo
▲ seizethecheese an hour ago

> a model that's more token efficient but more expensive per token can be cheaper for a given task.

True, but I'm skeptical that this is now overly used to ignore cost per token. Your comment inspired me to do a quick investigation. Looks like in my data, 92% of cost per task can be predicted by cost per token https://micjm.com/blog/cost-per-task-vs-token/

▲danielmarkbruce an hour ago | parent [-]

Definitely. Cost per task is nice in theory, pretty absurd in practice.

▲seizethecheese an hour ago | parent [-]

Well, it depends on the data source. Cost per task for benchmarks is absurd. Cost per task for real world usage, where the model identity is randomized (Like Arena or Agent Duel) seems better, though the tasks still are not going to be representative if you're not doing the same stuff as the battle participants.