| ▲ | XCSme 8 hours ago |
| Yup, they do quite poorly on random non-coding tasks: https://aibenchy.com/compare/minimax-minimax-m2-7-medium/moo... |
|
| ▲ | rmi_ an hour ago | parent | next [-] |
| Wild benchmark. Opus 4.6 is ranked #29, Gemini 3 Flash is #1, front of Pro. I'm not saying it's bad, but it's definitely different than the others. |
| |
| ▲ | XCSme an hour ago | parent [-] | | The main reason is that Claude models tend to ignore instructions. There is a failure example on the Methodology page. | | |
| ▲ | BoorishBears an hour ago | parent [-] | | > It is not my fault if Claude outputs something like "*1*, *1*", adding markdown highlighting, when most other models respect the required format correctly. Yuck. At that point don't publish a benchmark, explains why their results are useless too. - Edit since I'm not able to reply to the below comment: "I want structured output from a model that supports structured output but will not enable structured output, nor ask for an existing format like XML or JSON" is not really an interesting thing to benchmark, and that's reflected in how you have Gemini 2.5 Flash beating GPT-5.4. I really hope no one reads that list and thinks it's an AI leaderboard in any generalizable sense. | | |
| ▲ | XCSme an hour ago | parent [-] | | Why not? I described this in more detail in other comments. Even when using structured output, sometimes you want to define how the data should be displayed or formatted, especially for cases like chat bots, article writing, tool usage, calling external api's, parsing documents, etc. Most models get this right. Also, this is just one failure mode of Claude. |
|
|
|
|
| ▲ | usagisushi 5 hours ago | parent | prev | next [-] |
| Interesting benchmark. It is notable that Gemini-3-Flash outperforms 3.1 Pro. My experience using Flash via Opencode over the past month suggests it is quite underrated. Needless to say, benchmarks are limited and impressions vary widely by problem domain, harness, written language, and personal preference (simplicity vs detail, tone, etc.). If personal experience is the only true measure, as with wine, solving this discovery gap is an interesting challenge (LLM sommelier!), even if model evolution eventually makes the choice trivial. (I prefer Gemini 3 for its wide knowledge, Sonnet 4.6 for balance, and GLM-5 for simplicity.) |
|
| ▲ | wizee 6 hours ago | parent | prev | next [-] |
| It’s worth also comparing Qwen 3.5, it’s a very strong model. Different benchmarks give different results, but in general Qwen 3.5, GLM 5, and Kimi K2.5 are all excellent models, and not too far from current SOTA models in capability/intelligence. In my own non-coding tests, they were better than Gemini 3.1 flash. They’re comparable to the best American models from 6 months ago. |
| |
| ▲ | XCSme 5 hours ago | parent | next [-] | | I used qwen 3.5 plus in production, it was really good at instruction following and tool calling. | |
| ▲ | 5 hours ago | parent | prev [-] | | [deleted] |
|
|
| ▲ | comboy an hour ago | parent | prev [-] |
| Not really related, but does anybody know if somebody's tracking same models performance on some benchmarks over time? Sometimes I feel like I'm being A/B tested. |
| |
| ▲ | XCSme an hour ago | parent [-] | | Oh, I didn't think about this, that's a good idea. I also feel generally model performance changes over time (usually it gets worse). The problem with doing this is cost. Constsntly testing a lot of models on a large dataset can get really costly. |
|
