Remix.run Logo
▲ danielmarkbruce an hour ago

Definitely. Cost per task is nice in theory, pretty absurd in practice.

▲seizethecheese an hour ago | parent [-]

Well, it depends on the data source. Cost per task for benchmarks is absurd. Cost per task for real world usage, where the model identity is randomized (Like Arena or Agent Duel) seems better, though the tasks still are not going to be representative if you're not doing the same stuff as the battle participants.