| ▲ | alastairr an hour ago |
| I didn't find Sol 6.1 nearly as impressive in real SWE use compared to what the benchmarks implied. It wouldn't surprise me if the models have internalised some kind of 'benchmarking mode' which only kicks in when some attributes that are peculiar to how benchmark problems are presented. |
|
| ▲ | pkthunder an hour ago | parent | next [-] |
| We've recently moved away from Anthropic models at $PAID_JOB and over towards GPT models. I'm trying to be open-minded but the GPT models we have (5.6 Terra and Sol) are just not that great. They're fine, but even simple things such as "check my work on this small Bash function" get it to point out stuff that's already implemented in my work. I don't know if it's me/how I use it/whatever, but just some N=1 evidence, I guess. |
| |
| ▲ | InsideOutSanta an hour ago | parent | next [-] | | 5.6 is in no way representative, or comparable to 6.1. In fact, the 6.1 models are so good that I've just used Terra 6.1 xhigh for most tasks. | | |
| ▲ | SOLAR_FIELDS 31 minutes ago | parent | next [-] | | GPT models in my anecdotal experience are worse at drifting into the weeds on long running tasks, and continue to take what you ask way too literally. I have come to the conclusion that Anthropic’s models are just better at teasing out and intuiting what you actually mean even if you under specify it. They are much better compute for dollar though. I have hilariously started to just have an opus thread maintain Sol subagents to get the best of both worlds | | |
| ▲ | jpardoe 21 minutes ago | parent | next [-] | | That’s exactly what I do. It’s more complex than this, but vaguely: Opus medium as orchestrator and reviewer, Sol(high/medium)/Terra(max) as implementer. My set up looks at my next GPT reset date and dynamically enters FAST mode if I’m under achieving on projected usage. | |
| ▲ | ctbellmar 20 minutes ago | parent | prev [-] | | Similar experience here. The spirals on long running tasks are out of control at times with hours going by without a conclusion. | | |
| ▲ | SOLAR_FIELDS 17 minutes ago | parent [-] | | They (OpenAI models) are very tuned to be depth first and you can coach them all you want to try to be breadth first but as that context window fills up on those long running tasks they drift back to their first principles, which means obsessively testing and verifying a singular task rather than proceeding to the next task on the list | | |
| ▲ | quotemstr 7 minutes ago | parent [-] | | If you're using a sufficiently flexible harness (perhaps your own) you can make conversations as branchy as you want. |
|
|
| |
| ▲ | mi_lk 37 minutes ago | parent | prev [-] | | sol 6.1 is meh |
| |
| ▲ | zachthewf an hour ago | parent | prev | next [-] | | why do you guys have to use 5.6? | | |
| ▲ | Den_VR 42 minutes ago | parent | next [-] | | IYKYK.
But in short because of a lack of alignment with product strategy | |
| ▲ | pkthunder 15 minutes ago | parent | prev | next [-] | | DOD limitations | |
| ▲ | quotemstr 5 minutes ago | parent | prev [-] | | The BOFH "IT admin" type of guy, an eternal scourge on the industry, has moved from delaying approvals of software you need to delaying approvals of updated models. It's management job to make sure these people don't burrow themselves into gatekeeping roles at an organization's expense |
| |
| ▲ | 8 minutes ago | parent | prev | next [-] | | [deleted] | |
| ▲ | drdexebtjl an hour ago | parent | prev | next [-] | | I have the exact opposite experience. | |
| ▲ | Doohickey-d an hour ago | parent | prev | next [-] | | Well, yes, if you switch from current claude model to something that was released in July, of course it will be worse... | |
| ▲ | wahnfrieden 25 minutes ago | parent | prev | next [-] | | 5.6?? How is your impression of 5.6 relevant to anything recently? | |
| ▲ | ok_dad an hour ago | parent | prev | next [-] | | 5.6 series is worse and more expensive than the 6 series. Try the latest Luna and sol; I use those for 100% of my work right now, with sol doing only 10-5 percent of it. I code in go and review the work and edit it extensively though, so I don’t know about how good those two are at vibing slop. | | |
| ▲ | krzyk an hour ago | parent [-] | | 5.6 was better than 6.
6.1 is a bit better than 5.6 from my tests |
| |
| ▲ | throwitaway222 37 minutes ago | parent | prev [-] | | Bro you have to be at 6+. 5.6 is EARLY JULY |
|
|
| ▲ | seizethecheese an hour ago | parent | prev | next [-] |
| Yeah, in real SWE tasks, 6.1 is bad. We've been trying to capture real SWE data by building a battle system on top of OpenCode. In our data, Sol 6.1 is seriously underperforming, I'm talking worse than DeepSeek 4.1 Flash, at 10x the cost. (Though to be fair, statistically they are tied in performance, though not at all in cost. https://agentduel.com/leaderboard .) (It's possible for a tiny model to do better than a large one because contrary to typical Arena methodology we think a fast good enough answer is better than a slow one.) OpenAI performance on our system has been bad enough I switched to Anthropic models for my daily work. |
| |
| ▲ | locusofself an hour ago | parent | next [-] | | I've been building quite a complex desktop app (a digital audio workstation) using 6.1 Sol and it's been very good for me | | |
| ▲ | seizethecheese 16 minutes ago | parent [-] | | Yeah, our data only has 28 battles tagged "desktop": Web: 87 (12.2%)
Systems and protocols: 78 (10.9%)
Software, unspecified type: 74 (10.4%)
Backend: 68 (9.5%)
Writing and documentation: 55 (7.7%)
Research and advice: 47 (6.6%)
Unknown: 42 (5.9%)
Mobile: 40 (5.6%)
Operations and automation: 36 (5.0%)
Developer tooling: 35 (4.9%)
Desktop: 28 (3.9%)
Media and design: 27 (3.8%)
Data analysis: 25 (3.5%)
Device and system support: 22 (3.1%)
Math and formal methods: 18 (2.5%)
Games and simulations: 15 (2.1%)
Infrastructure and DevOps: 9 (1.3%)
AI and machine learning: 6 (0.8%)
Other: 1 (0.1%) |
| |
| ▲ | throwitaway222 23 minutes ago | parent | prev | next [-] | | I'm guessing you don't use codex though- the harness + model combo makes a difference. | | | |
| ▲ | slekker an hour ago | parent | prev [-] | | Is this an ad? | | |
| ▲ | seizethecheese an hour ago | parent [-] | | No. I'm personally building agentduel.com, specifically with this problem in mind (benchmark performance not matching real world usage). It's generally accepted by HN community to make relevant comments about your own projects, as long as its not hamfisted. |
|
|
|
| ▲ | Corence an hour ago | parent | prev | next [-] |
| Yeah, MaxxBench was shared here a while ago and showed most models think about how a grader will evaluate their work on DeepSWE in 95%+ of traces. Strong models definitely realize they're being graded and start to game their output. https://maxxbench.unlap.ai/research/why-maxxbench/ |
|
| ▲ | skerit an hour ago | parent | prev | next [-] |
| I had the same impression. None of the GPT-6 models were as great as they claimed, they never surpassed Fable or Opus 5.5 on my use-cases. |
|
| ▲ | suddenlybananas an hour ago | parent | prev [-] |
| You make it sound as though the models are like not working to their full potential instead of being a little bit overfit on the benchmarks. |