| ▲ | minimaxir 7 hours ago |
| Artificial Analysis index results are up, since multiple people are asking: https://artificialanalysis.ai/models/kimi-k3-1-pro It's 55, at 4th overall, beating both GPT-6 Astra and Fable 5.1. It's also faster than both. > When evaluating the Intelligence Index, it generated 150M tokens, which is very verbose in comparison to the median of 87M. |
|
| ▲ | thepasch 7 hours ago | parent | next [-] |
| > When evaluating the Intelligence Index, it generated 150M tokens, which is very verbose in comparison to the median of 87M. This is 100M fewer than Opus 5.5, 40M fewer than Fable 5.1, and about a third of Sonnet 5.5. In the top-6, it's 3rd, beaten by Muse Wonder at 140M and Astra at, impressively, just 60M. |
| |
| ▲ | jwolfe 6 hours ago | parent [-] | | That's versus Opus 5.5 max effort, no? Opus high has the closest cost and intelligence characteristics and generates far fewer output tokens per task than the one Kimi 3.1 Pro effort level we have data for. | | |
| ▲ | thepasch 6 hours ago | parent | next [-] | | Yes, max effort is what the displayed-by-default score on the index is measured by. At xhigh, Opus 5.5 outputs 100M tokens, but remains only 1 point ahead of K3.1 rather than 5. And Sonnet at xhigh outputs 110M tokens and trails K3.1 by 3 points rather than leading by 3, at the same price point per mtok. Opus 5.5 high trails K3.1 by 1 point, but outputs only 53M tokens, so it indeed comes out cheaper per-task. I think it's still remarkable that this is effectively an open model that matches the frontier for the first time in a long time, pretty much since DeepSeek R1. | | | |
| ▲ | TobTobXX 5 hours ago | parent | prev [-] | | I think that this is Kimi K3.1 Pro on max too. Unfortunately, AA doesn't seem to test for other effort levels for models not from Anthropic/OpenAI. |
|
|
|
| ▲ | mwigdahl 7 hours ago | parent | prev | next [-] |
| Muse Wonder is even higher, and there is no way in hell it's better in practice than either Astra or Fable. |
| |
| ▲ | jrflo 5 hours ago | parent [-] | | Muse Wonder Full* is higher, which by all accounts I've seen it's actually a good model | | |
|
|
| ▲ | SkiFire13 5 hours ago | parent | prev | next [-] |
| From that link: > reasonably priced when comparing to other models of similar price ??? |
| |
| ▲ | epolanski 2 hours ago | parent [-] | | It's saying that you'll do more work for the same $, but that's not anybody's experience with Kimi really. |
|
|
| ▲ | dom96 5 hours ago | parent | prev | next [-] |
| > beating both GPT-6 Astra and Fable 5.1. It's also faster than both. Very similar to my adversarial programming language (KillSwitch) benchmark[1], at least on pass rate, it is much slower though. 1 - https://bench.killswitch-lang.org |
| |
|
| ▲ | saberience 6 hours ago | parent | prev | next [-] |
| I don't trust these Artificial Analysis indexes and I'm not sure why anyone considers them authoritative, the site just sprung up a few months ago and now everyone quotes it? From my own use of the models I find their ranking to be totally inaccurate. |
| |
| ▲ | jrflo 5 hours ago | parent | next [-] | | They are the only site that really combines all benchmarks into a single place. And their intelligence index is pretty broad, including knowledge work and finance tasks as well as coding, so if your use case is narrow it may not match the general benchmarks | |
| ▲ | 0xcoffee 2 hours ago | parent | prev | next [-] | | Personally I do trust them, but I mainly look at the Intelligence Index vs. Time per Task, and
Intelligence Index vs. Cost per Task The per token charts do not provide meaningful information for me I also find the provider benchmarks fascinating: https://artificialanalysis.ai/models/deepseek-v4-1-flash-101... | |
| ▲ | augment_me 5 hours ago | parent | prev | next [-] | | Do you have an alternative? Its a benchmark collection. When a benchmark gets saturated, they remove it from the index and add new ones. I think lm-arena is a better concept that reflects the actual target users, but it only captures chat, and relies on users spending their time improving it. | | |
| ▲ | genneth 5 hours ago | parent [-] | | https://epoch.ai/eci?view=graph&tab=release-date is at least an alternative. They are slower to update, but entirely anecdotally I find the numbers are aligned with my personal observations a bit better than AA. But I am also highly aware this is some pretty confirmation-bias heavy ares of meta-benchmark picking... | | |
| ▲ | augment_me 5 hours ago | parent [-] | | If we ignore the older releases (sonnet 5, etc that exist in that one) and the new models epoch has not benchmarked, we end up with the same order within the noise brackets as in ArtificialAnalysis. I do trust epoch more than AA due to capital incentives, but in this case I don't think like there is a difference |
|
| |
| ▲ | sureMan6 5 hours ago | parent | prev [-] | | It's appealing to me because it's not just one benchmark but a summary of a collection of benchmarks. Also the current rankings have Opus on top by far and everyone's experience is that Opus is best by far so it's not massively inaccurate |
|
|
| ▲ | 7 hours ago | parent | prev | next [-] |
| [deleted] |
|
| ▲ | option 5 hours ago | parent | prev | next [-] |
| while this is an awesome model, I have a bridge to sell to folks who think (based on AA leaderboard) that this is better than Opus 5.5 |
|
| ▲ | red_hare 7 hours ago | parent | prev [-] |
| > When evaluating the Intelligence Index, it generated 150M tokens, which is very verbose in comparison to the median of 87M. So their base price is 1/5th of Opus 5.5 per token, but if you want "highspeed" it's 2x that and then I guess it generates 2x the number of tokens as well? That might suggest that Opus is competitively priced after all. |
| |
| ▲ | minimaxir 7 hours ago | parent | next [-] | | In an agentic pipeline, the output text is 2-5% of the total tokens used, so verbosity is a nonissue costwise. Although, higher verbosity could cause input to balloon faster and potentially cost more, but that would be captured by the cost-per-task metric. But as noted by other replies, it's still lower than Opus/Sonnet. | | |
| ▲ | jukea 5 hours ago | parent [-] | | You pay for chain of thought tokens as well. Ex: Imagine asking for a single number that takes 1h to compute | | |
| |
| ▲ | ctolsen 7 hours ago | parent | prev [-] | | Opus 5.5 is even more verbose. 260M on the same suite. |
|
