Remix.run Logo
▲ seizethecheese 2 hours ago

Does anyone really believe the cost data for benchmarks? Pretty sure all models will behave differently at this point when they get benchmark shaped questions.

(And I’m sitting on data showing Sol 6.1 and Qwen 3.8 Max cost roughly the same per task, for real tasks.)

▲eli an hour ago | parent | next [-]

The cost per task metric is accurate for those benchmark tasks. If your tasks don’t look like benchmark tasks..you should use your own benchmarks :)

▲XCSme an hour ago | parent [-]

Lol, I did that with https://aibenchy.com

▲killingtime74 2 hours ago | parent | prev [-]

I mean not believing intelligence score is reasonable but cost is just reporting what they spent?

▲seizethecheese 2 hours ago | parent [-]

I believe low cost but not high cost, since a model will work really hard when a question looks benchmark-like.

▲killingtime74 6 minutes ago | parent [-]

Ah, fair enough.