Remix.run Logo
▲ seizethecheese an hour ago

Yeah, in real SWE tasks, 6.1 is bad.

We've been trying to capture real SWE data by building a battle system on top of OpenCode. In our data, Sol 6.1 is seriously underperforming, I'm talking worse than DeepSeek 4.1 Flash, at 10x the cost. (Though to be fair, statistically they are tied in performance, though not at all in cost. https://agentduel.com/leaderboard .)

(It's possible for a tiny model to do better than a large one because contrary to typical Arena methodology we think a fast good enough answer is better than a slow one.)

OpenAI performance on our system has been bad enough I switched to Anthropic models for my daily work.

▲locusofself an hour ago | parent | next [-]

I've been building quite a complex desktop app (a digital audio workstation) using 6.1 Sol and it's been very good for me

▲seizethecheese 14 minutes ago | parent [-]

Yeah, our data only has 28 battles tagged "desktop":

Web: 87 (12.2%) Systems and protocols: 78 (10.9%) Software, unspecified type: 74 (10.4%) Backend: 68 (9.5%) Writing and documentation: 55 (7.7%) Research and advice: 47 (6.6%) Unknown: 42 (5.9%) Mobile: 40 (5.6%) Operations and automation: 36 (5.0%) Developer tooling: 35 (4.9%) Desktop: 28 (3.9%) Media and design: 27 (3.8%) Data analysis: 25 (3.5%) Device and system support: 22 (3.1%) Math and formal methods: 18 (2.5%) Games and simulations: 15 (2.1%) Infrastructure and DevOps: 9 (1.3%) AI and machine learning: 6 (0.8%) Other: 1 (0.1%)

▲throwitaway222 21 minutes ago | parent | prev | next [-]

I'm guessing you don't use codex though- the harness + model combo makes a difference.

▲slekker an hour ago | parent | prev [-]

Is this an ad?

▲seizethecheese 43 minutes ago | parent [-]

No. I'm personally building agentduel.com, specifically with this problem in mind (benchmark performance not matching real world usage). It's generally accepted by HN community to make relevant comments about your own projects, as long as its not hamfisted.