Remix.run Logo
▲ freakynit 3 hours ago

Those benchmarks can't possibly be real, right? Right? <insert padme 4-panel meme here>

▲logicchains 3 hours ago | parent [-]

At this rate the frontier labs will be Anthropic and DeepSeek, with OpenAI struggling to keep up.

▲A_D_E_P_T 2 hours ago | parent | next [-]

OpenAI's models -- and even Google's Argon -- have particular niches where they shine. For GPT-6/6.1, it's 3D modeling; Astra is hands-down the best at making .step files, Blender files, etc. For Argon, it's natural language writing and everything that hooks up to Google, e.g. search and maps-based itinerary planning.

Anthropic and DeepSeek (+ GLM) are definitely solving coding, though!

They're all quite different tools.

▲Kurtz79 2 hours ago | parent | next [-]

Artificial Analysis recently released a dedicated 3D benchmark, where Astra lags behind both Opus and Sonnet 5.5 (and yes, open weight models seem to be quite far behind).

https://artificialanalysis.ai/articles/aa-3d

▲A_D_E_P_T an hour ago | parent | next [-]

Interesting!

For me it's large files in Blender (in some cases >400MB), mostly, where Astra excels. I gave it a look, but I don't know if the benchmark encompasses that sort of thing.

.step files are simpler pieces of straightforward code, so it makes sense that Opus and Sonnet 5.5 do very well...

▲swiftcoder an hour ago | parent | prev [-]

its unfortunate they didn't include deepseek in the initial batch of benchmarks

▲ 2 hours ago | parent | prev [-]
[deleted]
▲freakynit 3 hours ago | parent | prev | next [-]

I believe these are the last sets of super-high-end models available to public (open or closed).. these companies probably will now start to withhold top-of-the-line models for themselves, governments, and their super-high-paying partners only.

▲stymaar 3 hours ago | parent | next [-]

News from a near future:

Here's How Distill APT Used McKinsey's Private Access To Top Anthropic Model To Help The Chinese Keep Up With The Us Top IA Labs Despite Access Restrictions

▲A_D_E_P_T 2 hours ago | parent | prev | next [-]

That hasn't been the way to bet. DeepSeek and the other Chinese firms show no signs of slowing down, so if OAI and Anthropic halt releases and "pace the frontier" they'll just fall behind. (And there's a chance that high-end AI will become fully commoditized, which suits DeepSeek just fine.)

Besides, Fable 5.5 is reportedly releasing within the next few days, probably on Tuesday...

▲girvo 30 minutes ago | parent [-]

China just announced tightened export controls on AI models, for what it's worth.

▲ReptileMan 2 hours ago | parent | prev [-]

We have the weights, we have the papers, nothing preventing development on this foundation.

▲slopinthebag 2 hours ago | parent | prev [-]

idk, sol 6.1 is both cheap and capable. i've been using it exclusively, i don't need to care about usage, and it's really smart.

yes maybe opus 5.5 is stronger and flash is cheaper, but 6.1 strikes a good balance imo.

▲computerex 41 minutes ago | parent | next [-]

OpenAI models don’t come close to competing with deepseek in cost quality ratio. I have a $20 plus plan and I run out of my 5h limit in literal minutes, every tick of the harness is like 10% of my 5h limit.

You can in contrast get far more done with $20 worth of deepseek api spend. No comparison.

▲ReptileMan 2 hours ago | parent | prev [-]

i have a feeling that Sol is just Astra low, but with lowest possible priority on the compute.

▲WASDx 2 hours ago | parent [-]

I believe it's Astra quantized. It was briefly named astra-lite.