| |
| ▲ | XCSme 3 hours ago | parent [-] | | I have created various questions/tests and put the models through the same tests. I record whether the answers are correct, and the generation stats (costs, latencies, tokens used, etc.). I have no idea why the Gemini models do so well. I have recently added new tests, whose sole purpose was to find some cases on which Gemini 3 Flash fails (I don't like cherry-picking models or tests, but I also find it strange Gemini Flash models leading in accuracy). I made a more complex coding/tool-usage test, that I expected it to fail, it did fail it once locally in my debug tests, but when I finalized the test and ran the entire testing suite for all models, somehow Gemini 3 Flash still got it right... Gemini models are REALLY intelligent (and they are actually my favorite model to use via the chat app to ask questions), but they somehow fail in real-word coding tasks where they have to modify files, check results, debug, etc. My tests harness provides a lot of mock data, and limits the number of actions a model can choose from. I am starting to think that maybe the models are not bad, just that the coding harness are not optimized for those type of models, and Google doesn't really provide their own "Codex". | | |
| ▲ | jdthedisciple 2 hours ago | parent | next [-] | | Interesting, well it'd be interesting to check out some individual examples where Gemini beat the others. Also, would be great if you could add GPT 5.6 Sol XHigh and Fable 5 High as well, just to see if at least those beat Gemini which is currently your #1. | | |
| ▲ | XCSme 2 hours ago | parent [-] | | I don't like to divulge tests, but one of them is a chess puzzle. > would be great if you could add GPT 5.6 Sol XHigh and Fable 5 High as well I would like too, but I avoided them for several reasons: 1) Cost - this is a hobby project, those models would cost tens of dollars for each benchmark run, multiply this by tens or hundreds of models and ... 2) Time - the high models are already taking a really long answer to respond (5-10minutes per question). I run each question with 3 repeats (run the same test three times), so it would take 30 minutes per test. If I change my tests, methodology, or add a new test, it would take a really long time to run the benchmark. Also, I like having results immediately when a new model is released, now I can post within 30 minutes of a model's release the benchmark results. 3) High reasoning usually does WORSE on most tests - if you look at the leaderboard, it's sometimes counter-intuitive, but models with high or max reasoning usually do worse than medium and low. This is because the questions are quite targeted/direct, and the models overthink the question and miss the solution. Or the long thinking context makes them perform poorly. The generation tasks (SVGs/HTML animation) are usually better with longer reasoning, but short code fixes, trivia questions, puzzles, etc. are answered by low/med reasoning with more accuracy in general Also, Fable is borderline un-testable, it refuses to answer many questions, so it scores poorly anyway. Gemini scores 21/22 because it answers all tests, and it does them correctly, consistently. The only failed test is I think because it miscounted the lines in a file, when responding on which line the bug was in a code snippet. |
| |
| ▲ | XCSme 2 hours ago | parent | prev [-] | | Oh, and I've also added weights to different categories, so Coding and Tool usage categories influence the score more. This done both to better account for how most people are being used, and also to reduce Gemini's dominance in general/domain specific knowledge. So yes, Gemini models are at the top, even if I actually (not proud of it) tried to make tests that actually favour other coding-focused models. |
|
|
