Remix.run Logo
▲ intenex an hour ago

Performance is insane for the cost if those benchmarks hold up. Looks like they did time this for right before Anthropic's IPO. Curious what Anthropic releases in response, and how these do on real world tasks.

Feels like we're way past the pelican benchmark being useful at this point. Anyone have a better benchmark for a quick vibe check on how this compares to Fable 5.2 and Opus 5.5?

Edit: Wow, interesting very negative vibe check here: https://x.com/mattshumer_/status/2122752192226406421?a=fVRt3...

▲mlinsey an hour ago | parent | next [-]

If you have the quota or budget for it, I recommend vibe-coding your own personal benchmark. I have a suite of 8 projects that I made with models this year, and replay them - give the new model a checkout of the repo at that point, and the project spec. For the design task, I have it throw up a blind voting page for me to quickly judge, and for implementation I have a review agent do a general score and also hunt for problems/edge cases that I really found.

It's a lot more imprecise than the public benchmarks, but it's tailored to my own work and definitely isn't being benchmaxxed by the model developers. I was able to tell really quickly that Opus 5.5 delivered on the hype, and that some other releases (a couple of the more recent open releases, and pre-Carbon gemini models) were not as good for me as public benchmarks made them look.

(no results for either 6.2 yet, will take another ~6-8 hours, but I encourage people reading this with spare tokens to just make their own like I did instead of listen to me)

▲seizethecheese 30 minutes ago | parent | next [-]

Your personal voting system sounds a lot like what we're building here. https://github.com/bottomless/agent-duel. We even have the LLM judge that helps parse out any differences. Related to this discussion, my personal Elo board based on my own votes has Sol 6.1 dead last:

Claude Opus 5.5 1160 (1027–1334) GPT-6 Astra 1114 (932–1330) Muse Wonder 1063 (918–1220) MiMo-V2.6 Flash 1053 (925–1187) Kimi K3.1 Pro 1037 (870–1216) Gemini Carbon 1021 (762–1281) Claude Fable 5.2 975 (682–1239) DeepSeek V4.1 Flash 952 (799–1095) Grok 4.7 927 (782–1058) GPT-6.1 Sol 910 (677–1096)

▲grey-area an hour ago | parent | prev | next [-]

This is much more interesting IMO than any public benchmark. They are definitely targeting those benchmarks and model capability does seem to have plateaued somewhat.

▲behat 21 minutes ago | parent | prev | next [-]

This is great, and I have been setting up infra to do this very thing. How many sessions to you replay typically to get a reasonably confident estimate? Assuming each project in your suite was built over multiple sessions.

▲Havoc an hour ago | parent | prev | next [-]

What type of tasks did you use for this? Evaluation?

Last time I did this I found it quite hard to come up with tasks that have clear evaluation outcomes

▲bertmuir 30 minutes ago | parent [-]

Have it compare to the real work you've done in a sample of your previous sessions. Agents are good at simulating completing the work following newer harnesses, heuristics etc. and comparing time, tokens, quality, pitfalls.

▲srcreigh an hour ago | parent | prev [-]

please update !

▲alastairr an hour ago | parent | prev | next [-]

I didn't find Sol 6.1 nearly as impressive in real SWE use compared to what the benchmarks implied. It wouldn't surprise me if the models have internalised some kind of 'benchmarking mode' which only kicks in when some attributes that are peculiar to how benchmark problems are presented.

▲pkthunder an hour ago | parent | next [-]

We've recently moved away from Anthropic models at $PAID_JOB and over towards GPT models. I'm trying to be open-minded but the GPT models we have (5.6 Terra and Sol) are just not that great. They're fine, but even simple things such as "check my work on this small Bash function" get it to point out stuff that's already implemented in my work. I don't know if it's me/how I use it/whatever, but just some N=1 evidence, I guess.

▲InsideOutSanta an hour ago | parent | next [-]

5.6 is in no way representative, or comparable to 6.1. In fact, the 6.1 models are so good that I've just used Terra 6.1 xhigh for most tasks.

▲SOLAR_FIELDS 42 minutes ago | parent | next [-]

GPT models in my anecdotal experience are worse at drifting into the weeds on long running tasks, and continue to take what you ask way too literally. I have come to the conclusion that Anthropic’s models are just better at teasing out and intuiting what you actually mean even if you under specify it.

They are much better compute for dollar though. I have hilariously started to just have an opus thread maintain Sol subagents to get the best of both worlds

▲jpardoe 32 minutes ago | parent | next [-]

That’s exactly what I do. It’s more complex than this, but vaguely: Opus medium as orchestrator and reviewer, Sol(high/medium)/Terra(max) as implementer. My set up looks at my next GPT reset date and dynamically enters FAST mode if I’m under achieving on projected usage.

▲ctbellmar 31 minutes ago | parent | prev [-]

Similar experience here. The spirals on long running tasks are out of control at times with hours going by without a conclusion.

▲SOLAR_FIELDS 29 minutes ago | parent [-]

They (OpenAI models) are very tuned to be depth first and you can coach them all you want to try to be breadth first but as that context window fills up on those long running tasks they drift back to their first principles, which means obsessively testing and verifying a singular task rather than proceeding to the next task on the list

▲quotemstr 19 minutes ago | parent [-]

If you're using a sufficiently flexible harness (perhaps your own) you can make conversations as branchy as you want.

▲mi_lk an hour ago | parent | prev [-]

sol 6.1 is meh

▲zachthewf an hour ago | parent | prev | next [-]

why do you guys have to use 5.6?

▲Den_VR an hour ago | parent | next [-]

IYKYK. But in short because of a lack of alignment with product strategy

▲pkthunder 26 minutes ago | parent | prev | next [-]

DOD limitations

▲quotemstr 17 minutes ago | parent | prev [-]

The BOFH "IT admin" type of guy, an eternal scourge on the industry, has moved from delaying approvals of software you need to delaying approvals of updated models. It's management job to make sure these people don't burrow themselves into gatekeeping roles at an organization's expense

▲drdexebtjl an hour ago | parent | prev | next [-]

I have the exact opposite experience.

▲ 19 minutes ago | parent | prev | next [-]
[deleted]
▲Doohickey-d an hour ago | parent | prev | next [-]

Well, yes, if you switch from current claude model to something that was released in July, of course it will be worse...

▲wahnfrieden 36 minutes ago | parent | prev | next [-]

5.6?? How is your impression of 5.6 relevant to anything recently?

▲ok_dad an hour ago | parent | prev | next [-]

5.6 series is worse and more expensive than the 6 series. Try the latest Luna and sol; I use those for 100% of my work right now, with sol doing only 10-5 percent of it. I code in go and review the work and edit it extensively though, so I don’t know about how good those two are at vibing slop.

▲krzyk an hour ago | parent [-]

5.6 was better than 6. 6.1 is a bit better than 5.6 from my tests

▲throwitaway222 an hour ago | parent | prev [-]

Bro you have to be at 6+. 5.6 is EARLY JULY

▲seizethecheese an hour ago | parent | prev | next [-]

Yeah, in real SWE tasks, 6.1 is bad.

We've been trying to capture real SWE data by building a battle system on top of OpenCode. In our data, Sol 6.1 is seriously underperforming, I'm talking worse than DeepSeek 4.1 Flash, at 10x the cost. (Though to be fair, statistically they are tied in performance, though not at all in cost. https://agentduel.com/leaderboard .)

(It's possible for a tiny model to do better than a large one because contrary to typical Arena methodology we think a fast good enough answer is better than a slow one.)

OpenAI performance on our system has been bad enough I switched to Anthropic models for my daily work.

▲locusofself an hour ago | parent | next [-]

I've been building quite a complex desktop app (a digital audio workstation) using 6.1 Sol and it's been very good for me

▲aatd86 5 minutes ago | parent | next [-]

Yeah my main models are OpenAIs... I am trying Opus right now because I dont have any credit anymore but from experience, it was never as thorough as openai's models. It can be genius and unblock at times, but I've never been able to trust it.

▲seizethecheese 27 minutes ago | parent | prev [-]

Yeah, our data only has 28 battles tagged "desktop":

Web: 87 (12.2%) Systems and protocols: 78 (10.9%) Software, unspecified type: 74 (10.4%) Backend: 68 (9.5%) Writing and documentation: 55 (7.7%) Research and advice: 47 (6.6%) Unknown: 42 (5.9%) Mobile: 40 (5.6%) Operations and automation: 36 (5.0%) Developer tooling: 35 (4.9%) Desktop: 28 (3.9%) Media and design: 27 (3.8%) Data analysis: 25 (3.5%) Device and system support: 22 (3.1%) Math and formal methods: 18 (2.5%) Games and simulations: 15 (2.1%) Infrastructure and DevOps: 9 (1.3%) AI and machine learning: 6 (0.8%) Other: 1 (0.1%)

▲throwitaway222 34 minutes ago | parent | prev | next [-]

I'm guessing you don't use codex though- the harness + model combo makes a difference.

▲seizethecheese 12 minutes ago | parent [-]

There was a post about that today on HN but it looks largely to be bogus. I wrote a blog post about it, but not linking here to avoid too much self promotion in this thread. Funny enough my conclusion was that OpenAI models seem to be the only ones that actually do have an own-harness interaction, hah.

▲slekker an hour ago | parent | prev [-]

Is this an ad?

▲seizethecheese an hour ago | parent | next [-]

No. I'm personally building agentduel.com, specifically with this problem in mind (benchmark performance not matching real world usage). It's generally accepted by HN community to make relevant comments about your own projects, as long as its not hamfisted.

▲WarmWash 4 minutes ago | parent | next [-]

Using non-native harnesses sort of throws a wrench in things. I pretty sure your benchmark would look much different if outputs came from their intended harness.

▲arcanemachiner 4 minutes ago | parent | prev [-]

It certainly does get tiresome though.

▲arcanemachiner 5 minutes ago | parent | prev [-]

Welcome to HN.

BTW check out my totally real project that helps you detect native advertising on your favorite platforms:

https://github.com/blah/blah

▲Corence an hour ago | parent | prev | next [-]

Yeah, MaxxBench was shared here a while ago and showed most models think about how a grader will evaluate their work on DeepSWE in 95%+ of traces. Strong models definitely realize they're being graded and start to game their output.

https://maxxbench.unlap.ai/research/why-maxxbench/

▲skerit an hour ago | parent | prev | next [-]

I had the same impression. None of the GPT-6 models were as great as they claimed, they never surpassed Fable or Opus 5.5 on my use-cases.

▲aleqs 10 minutes ago | parent | prev | next [-]

We know anthropic nerfs models all the time, openai probably does the same. There is no reason to believe the benchmarks/numbers they release are even measured with the same effective model that customers would get.

▲suddenlybananas an hour ago | parent | prev [-]

You make it sound as though the models are like not working to their full potential instead of being a little bit overfit on the benchmarks.

▲platinumrad 19 minutes ago | parent | prev | next [-]

Shumer is a known scammer (Reflection 70B, among other things). Why does he have any credibility?

▲tiborsaas 4 minutes ago | parent | next [-]

Anybody who says stuff like "They were essentially unusable." without any context or proof is going to my permaban on credibility. Even 1 year old models are perfectly usable let alone frontier ones.

▲wilkystyle 4 minutes ago | parent | prev [-]

Not to mention he didn't actually provide any reason why the models are bad. When pressed in the comments and asked what was bad, he said "everything"...

▲revolvingthrow an hour ago | parent | prev | next [-]

> Looks like they did time this for right before Anthropic's IPO.

Would it be in OAI's interest to kick the legs from under Anthropic? Granted I don't think they can, but a shot across the bow can sometimes be more damaging that one expects (I think FTX collapsed due to one? Or maybe I'm misremembering).

I always assumed that if one of the big AI IPOs fail hard enough the neoclouds start to go bust, which causes a chain reaction that probably causes both ANT and OAI to collapse.

▲kridsdale3 21 minutes ago | parent | next [-]

I guess it's a good idea for them to wreck the appeal for their own employees to leave for the competitor.

▲arscan an hour ago | parent | prev [-]

My reaction as well - at the very least now is probably not the best time to knock down the realized valuation of a peer when they are effectively setting the market value of the whole category of ai companies, including yours… right?

▲stbenjam 28 minutes ago | parent | prev | next [-]

My vibe bench is no dependency HTML bowls of ramen https://bitbin.de/ramen-bench/

Opus 5.5 Max is my favorite, but some of the Sol/Astra 6.2 ones are pretty good.

Edit: some folks reported crashes, some of these are resource intensive. If you're able to share any debug info with me I'd appreciate it (OS, browser, which bowl, etc)

▲dava96 24 minutes ago | parent | next [-]

just a fyi this crashed my browser

▲stbenjam 19 minutes ago | parent | next [-]

Which browser / version / os? And was it any particular bowl that crashed it?

It works fine for me on Chrome, Safari and Firefox. However, some models have produced crazy resource intensive ramens before. I audit them now before pushing to make sure they're not too resource intensive

▲kridsdale3 20 minutes ago | parent | prev | next [-]

data point: no trouble seeing it on Chrome on MacOS (whatever the last one before the liquid glass update) on M5.

▲NiloCK 22 minutes ago | parent | prev [-]

Seconded.

▲Pxtl 8 minutes ago | parent | prev | next [-]

Poking around models crashed the tab in Firefox on windows.

▲fullstackchris 4 minutes ago | parent | prev [-]

Wild. The opus ones literally look photo realistic... (Want to declare I am NOT a payed anthropic spammer, but look at those for yourself)

Also confirms grok is absolute garbage

▲stbenjam 2 minutes ago | parent [-]

Yep, seeing Opus results convinced me that maybe Anthropic is right on not neeeding an image gen model. It was very slow though.

▲bob1029 7 minutes ago | parent | prev | next [-]

gpt-6.1-terra is already beyond end game for my client's business. They don't require anything more powerful.

Cost is the most important factor now.

▲xtracto 41 minutes ago | parent | prev | next [-]

>Feels like we're way past the pelican benchmark being useful at this point. Anyone have a better benchmark for a quick vibe check on how this compares to Fable 5.2 and Opus 5.5?

Generate an SVG with complete and detailed graphical instructions on how to fold an origami pelican riding a bicycle

(or any other complex origami object). The fact that it has to teach [you] something makes it pretty interesting.

▲teknologist 5 minutes ago | parent | prev | next [-]

Beware of the paid influencers who don't disclose their affiliations in this space. Theo is another one who now talks smack about anything that isn't Claude. They bend reality to fit a narrative.

▲the_duke an hour ago | parent | prev | next [-]

> Looks like they did time this for right before Anthropic's IPO.

While both Anthropic and OpenAI have been playing games with model announcement timings, this time I don't think it's related.

These are incremental bumps, not big new models that have a huge impact.

▲i_have_an_idea 20 minutes ago | parent | prev | next [-]

An Anthropic shareholder is very negative about an Anthropic-beating model release, right before the Anthropic IPO? Interesting, yes :)

▲6thbit 41 minutes ago | parent | prev | next [-]

They're going after anthropic's market through cheaper intelligence per token.

I wouldn't be surprised if they released 6.2 luna with the most attractive positioning on the pareto frontier right as the IPO happens.

▲karimf an hour ago | parent | prev | next [-]

I'm very curious about this as well.

Recursive self-improvement (RSI) has been a very hot topic recently, and it could be the biggest factor that decide Anthropic's IPO.

I wonder if the general public can infer whether RSI has started based on the recent performance gain of the frontier model.

▲HarHarVeryFunny 35 minutes ago | parent | next [-]

RSI just means using the LLM as part of the development process, which they are already doing. I doubt any of these companies are going to fire all their developers, so there is no point development is going to be 100% automated - for the foreseeable future it will always be a mix of humans using LLMs.

So, you can say they are doing "RSI" (aka using tools to automate development) already, and will continue to do so.

▲antupis an hour ago | parent | prev | next [-]

Why IPO if you have RSI and possible ASI?

▲AlexCoventry 27 minutes ago | parent | next [-]

So you can scale up. RSI is going to be compute-bound. ML experiments cost compute. Also, if you can make a genius in a data center, why not make a hundred/a million geniuses?

▲esafak 44 minutes ago | parent | prev [-]

So investors can cash out on the time frame they promised investors.

▲ppierre an hour ago | parent | prev [-]

Yeah we see some Really Stupide Intervention from those who drank the Kool aid. IAS2[] start to be a real thing.

[] IA Stockholm Syndrome.

▲ijustlovemath an hour ago | parent | prev | next [-]

I've been having fun with having it build interactive Three.js demos of devices, with setup screen, a few different operational modes, mechanical movements etc

▲aleqs 12 minutes ago | parent | prev | next [-]

None of these benchmarks/marketing numbers from openai or anthropic have ever held up.

Now that we know anthropic nerfs models after release, and without any notice/openess, there is no reason to even think these benchmarks they release are even done with the same model/intelligence that actual customers would get.

▲crimsoneer 26 minutes ago | parent | prev | next [-]

I'm just running it on my Blood Bowl Bench, holding it's own against the reigning champ Argon so far but early days...

https://botbowlbench.com/

▲ranyume an hour ago | parent | prev | next [-]

You're looking for a vibe check?

▲behnamoh an hour ago | parent | prev [-]

> Feels like we're way past the pelican benchmark being useful at this point.

It was neither a benchmark nor a good one at that. Just a blogger's curiosity that grew out of control.