| ▲ | maplet 2 hours ago | |||||||||||||||||||||||||||||||
It scores a 43 on Artificial Analysis with an average cost of $3.27 [1]. For reference, GPT-6.1 Sol (medium) gets a 46 with an average of $0.36. | ||||||||||||||||||||||||||||||||
| ▲ | esperent 2 hours ago | parent | next [-] | |||||||||||||||||||||||||||||||
The table on that page shows gpt 6.1 sol (max) at $1.27 per task and it's the only model cheaper than Qwen 4 max. By comparison opus 5.5 (max) is $10.01. If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking. | ||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||
| ▲ | seizethecheese 2 hours ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||
Does anyone really believe the cost data for benchmarks? Pretty sure all models will behave differently at this point when they get benchmark shaped questions. (And I’m sitting on data showing Sol 6.1 and Qwen 3.8 Max cost roughly the same per task, for real tasks.) | ||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||
| ▲ | scotty79 2 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||
Looks like a slightly weaker, cheaper Kimi. | ||||||||||||||||||||||||||||||||
| ▲ | thefourthchime 12 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||
[dead] | ||||||||||||||||||||||||||||||||
| ▲ | augment_me 14 minutes ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||
Artificial Analysis is a VC-backed firm and it's unclear what the incentives are there. They have altered their benchmark multiple times in the last 4 weeks (v5 in https://artificialanalysis.ai/articles/artificial-analysis-i...) and the last 5.0.1 change https://artificialanalysis.ai/methodology/intelligence-bench... pushed up Anthropic's models a bit, you can see that v5.0.0 set on the first link Opus 5.5 has a lower score than in v5.0.1, and v5.0.1 was released just before Fable dropped. Some part of me believes that the benchmark mix was put together to favor Anthropic's models. | ||||||||||||||||||||||||||||||||
| ▲ | montroser 2 hours ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||
Yeah, it's in the same ballpark as Kimi 3.1. Both of which are obviously open weight, so that's at least a big differentiator from Sol. Let's hope against hope that the much smaller flash model might beat deepseek v4.1 flash and run for as cheap. | ||||||||||||||||||||||||||||||||
| ▲ | jrflo an hour ago | parent | prev [-] | |||||||||||||||||||||||||||||||
It looks pretty bad on those benchmarks. Is it possible that this is due to the recent crackdown on distillation? | ||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||