| ▲ | mattlondon 8 hours ago |
| Currently top at https://deepswe.datacurve.ai - beating Opus 5! https://artificialanalysis.ai/models/gemini-3-8-flash shows an intelligence score of 59, the same as Opus 5 medium! Wow - for a flash model this seems to benchmark powerfully. Remains to be seen what it is like to use. |
|
| ▲ | theHocineSaad 7 hours ago | parent | next [-] |
| As of writing this comment, Claude Opus 5 has an intelligence score of 63, not 59 (it's not the same as Gemini 3.8 Flash). With a score of 59, Gemini 3.8 Flash is in eighth place, falling behind even Grok 4.6, Kimi k3, and GLM 5.3. https://imgur.com/a/BMOJBED |
| |
| ▲ | Squarex 7 hours ago | parent | next [-] | | They are all much larger and more expensive models. Google does not have a frontier model right now, but for cheap ones, they are better than event the chinese models now. | | |
| ▲ | pietz 7 hours ago | parent | next [-] | | That's not being debated here. The initial reported numbers were false and this was simply pointed out. You're changing the subject. | | |
| ▲ | mattlondon 7 hours ago | parent | next [-] | | Opus 5 medium has the same score as 3.8 flash on artificial analysis intelligence index. Are you implying Google or Artificial Analysis are reporting false numbers? What's your source? | | |
| ▲ | asdfologist 7 hours ago | parent [-] | | BTW you're comparing 3.8 flash high to opus 5 medium. 3.8 flash medium scores lower. | | |
| ▲ | WarmWash 5 hours ago | parent [-] | | Flash models are on the order of 1/10th the size of Opus models, so some flex in the thinking level is fair. | | |
| ▲ | Topfi 2 hours ago | parent [-] | | Flash is just a name with no defined or consistent meaning even within labs, let alone between them. Considering both are closed weight, there is no way to truly assess how big the size delta between the two is. Then again, who cares about size, performance and end-to-end speed+cost are what matters along with task adherence, task assessment and so on. Model size also can not be inferred by tokens/sec for a multitude of reasons, but to showcase two examples, Opus 5 and Sonnet 5, as well as Gemini 3.1 Pro Preview and 3.1 Flash have each very comparable output speeds when using the same deployment as a basis for comparison, despite it being very likely that within their generation, the former are larger than the latter. Feel the need to mention this, as I unfortunately stumble upon so many poorly reasoned, speculative hype post trying to infer model size via utterly unreliable metrics, not based in actual data. It’s like comments below arguing about the reasoning levels not normalized to some metric (like cost, output token amount or duration) but just the labels or high, max, medium, etc. Those mean almost nothing even when comparing models based on the same pretrain (just compare GPT-5.4 to GPT-5.2), they mean less than nothing comparing different labs releases. |
|
|
| |
| ▲ | duplessitous 5 hours ago | parent | prev [-] | | > [...] shows an intelligence score of 59, the same as Opus 5 medium! Nothing here is false, you are simply confused. You either didn't read what they wrote in its entirety or decided to reinterpret what they did write. | | |
| |
| ▲ | porphyra 3 hours ago | parent | prev [-] | | Better than even the Chinese models? That's a difficult-to-quantify, extremely rapidly moving target. Just today, Qwen 3.8 Max 0902 came out with a huge improvement over the previous Qwen 3.8 Max. |
| |
| ▲ | kamranjon 6 hours ago | parent | prev | next [-] | | They said Opus 5 medium - which does have an intelligence score of 59 (you have to select it manually from the dropdown to see it) | |
| ▲ | anthonyrstevens 6 hours ago | parent | prev [-] | | That 63 score is for Max. The OP specified medium. |
|
|
| ▲ | markasoftware 8 hours ago | parent | prev | next [-] |
| On artificial analysis it's only equal to opus 5 medium effort. Opus 5 max scores 63. Further, opus 5 medium outputs 4x fewer tokens to achieve the same result, negating a lot of the speed difference. |
| |
| ▲ | irishcoffee 7 hours ago | parent [-] | | A comparison to an artificial score and a comparison to “the same task” These folks must laugh themselves to sleep. This whole industry hoodwinked the masses. It’s impressive. | | |
|
|
| ▲ | WarmWash 8 hours ago | parent | prev | next [-] |
| The benchmark also doesn't include speed. You almost think something has gone wrong when using it because it returns full responses so incredibly fast. |
| |
| ▲ | sotix 24 minutes ago | parent | next [-] | | This one uses that as a priority weight: https://winstonrc.github.io/ai-coding-agents-leaderboard/ | |
| ▲ | scrlk 8 hours ago | parent | prev [-] | | Not just speed, also reliability. IME, Gemini's speed and quality doesn't degrade badly during weekday working hours compared to OAI, and especially Anthropic. | | |
| ▲ | ford 7 hours ago | parent [-] | | I've had Gemini model API use degrade the most out of OAI/Anthropic/Google (often "over capacity" vs true failures) Not sure on consumer/product use though | | |
| ▲ | scrlk 7 hours ago | parent [-] | | That's interesting to hear. I should have added that I use Gemini through Google AI Studio as my general chat model, which probably explains our wildly different experiences. |
|
|
|
|
| ▲ | onlyrealcuzzo 8 hours ago | parent | prev | next [-] |
| The rumor is that 3.9 is an equal improvement in all directions, and that it should be another fast follow on like 3.7 and 3.8 were. It's almost across the board better than Terra at less than half the price. 3.9 is likely to approach Sol at the 1/10th the price. Hopefully OpenAI releases Astra first, and it's not only better than Sol but significantly cheaper, too. |
| |
|
| ▲ | bertili 7 hours ago | parent | prev | next [-] |
| A fifth of the cost of Opus 5! Google is certainly pushing the completion with this. |
| |
| ▲ | abirch 7 hours ago | parent [-] | | Gemini hasn't failed me for personal usage yet. I haven't had the opportunity to use it at work. | | |
| ▲ | panarky 7 hours ago | parent | next [-] | | I've been using 3.7 Flash to audit the work of Opus High, and Flash finds lots of subtle and insidious defects even while all the unit tests are green. Then I tell Opus to read the audit report and implement what it agrees with. Flash is really good at this, and it is blazing fast in Antigravity CLI. Easily 10x faster than Opus. Can't wait to try 3.8 Flash. If it's good enough, maybe I'll switch Flash to primary and make Opus the auditor. | | |
| ▲ | prodigycorp 7 hours ago | parent | next [-] | | it's very fast but it still doesnt come close to 5.6 sol, at least for me, in terms of gathering the context necessary to do extensive changes. | |
| ▲ | porridgeraisin 7 hours ago | parent | prev [-] | | Yeah the speed in agy cli is amazing. Whole files get written and "py_compile"d in a single blink of the eye its crazy. In india, my telco gives me google ai pro for free. And agy with flash goes a long way. |
| |
| ▲ | MaxikCZ 7 hours ago | parent | prev [-] | | Idk, was building/maintaining simple esp32 control program with antig/opus. After last update it defaulted to gflash3.7. I pasted an email requesting 2 changes into the chat prompt, it did one and took me 4 turns to get that one right. |
|
|
|
| ▲ | ttul 8 hours ago | parent | prev | next [-] |
| Crushing it on DeepSWE is a very big deal. Excited to give this a try. |
| |
| ▲ | pietz 7 hours ago | parent | next [-] | | I know everyone is benchmaxxing but this one feels one step too far. Doesn't DeepSWE have both public and private tasks? I'd love to see the diff here. It looks more like Google execs losing their mind and pressuring researchers to put DeepSWE directly into the training set. | |
| ▲ | re-thc 7 hours ago | parent | prev [-] | | > DeepSWE is a very big deal It's clearly been "dealt with" already. When it launched we had interesting gaps and definitely differences. Now every new release is "crushing it". | | |
| ▲ | ttul 5 hours ago | parent [-] | | Will look forward to the "feel" of the model in real testing. But I agree that these benchmarks do get "dealt with" rapidly. That's a shame, but I guess it's the times we live in. |
|
|
|
| ▲ | Gecko4072 8 hours ago | parent | prev | next [-] |
| Google - we're so back |
| |
| ▲ | oceanplexian 8 hours ago | parent [-] | | Only 1 point behind the Chinese SOTA from two months ago. | | |
| ▲ | nolok 5 hours ago | parent | next [-] | | If you care about points sure, but personnaly I care about price, performance, speed and reliability | |
| ▲ | roosterIllusi0n 7 hours ago | parent | prev [-] | | I had qwen 3.8 3bit model drop into chinese on long runs. I had to remind it to use english. Its still better than every gemma model I tried. Gemma deleted files on a harddrive to make space when there was over 2TB free. For long runs, gemma is useless. |
|
|
|
| ▲ | kimjune01 7 hours ago | parent | prev | next [-] |
| deepswe is public and can be considered contaminated. |
|
| ▲ | sunaookami 8 hours ago | parent | prev | next [-] |
| >shows an intelligence score of 59, the same as Opus 5! ...on Medium reasoning. Claude Opus 5 (high) is the default in e.g. Claude Code and scores 61. Still very impressive. |
|
| ▲ | satvikpendem 8 hours ago | parent | prev | next [-] |
| We'll see about that. I suspect benchmaxxing as all the labs do as I haven't found Gemini models to be nearly as good in agentic engineering compared to Claude or GPT models. |
| |
| ▲ | NitpickLawyer 7 hours ago | parent | next [-] | | If anything, gemini models are the least benchmaxxed out of any lab, IMO. | |
| ▲ | onlyrealcuzzo 8 hours ago | parent | prev [-] | | And the benchmarks agreed with you... until now. So, yes, maybe it's still not - but this would be the only time it would be highly suspicious / obvious benchmaxxing / obviously bad benchmarks. |
|
|
| ▲ | WhitneyLand 7 hours ago | parent | prev | next [-] |
| There are important gaps in that hot take. For example, it's not even close to Opus 5 on Terminal-bench 4.0, 19.1% vs. 51.8%. |
|
| ▲ | jrflo 7 hours ago | parent | prev | next [-] |
| sidenote, but wow sonnet 5 is shockingly bad on this benchmark. |
| |
| ▲ | notatoad 7 hours ago | parent [-] | | sonnet 5 is bad by almost any metric. anthropic really needs something to address the cheaper end of the market before they get left behind. Sonnet 5 sucks, and Haiku hasn't been updated in a year. meanwhile we've got gemini flash, luna, and GLM5.3 all delivering 90% of the performance for a small fraction of the cost. paying $25/mTok is going to start looking pretty silly soon. |
|
|
| ▲ | pkos98 7 hours ago | parent | prev [-] |
| Wait a week with your judgement - most likely, Google is just bench-maxing very hard.
If you look at the previous Flash models and the announcement on Google I/O, it was an absolute disaster. Reality diverged very much from the marketing (supposedly great benchmarks). |
