Sure about Dario (and all billionaire) weirdness, but no gains if you are a skilled senior is well, very far out in our experience (our company is 30 years old with mostly the original employees and founders): what we deliver now at the speed and quality we deliver it would have been impossible 10 years ago with our team size of skilled seniors. We replaced all the commercial products our clients and ourselves used with our own, giving us millions more revenue and profit with the upselling and efficiency benefits. We work for regulated clients: our code is reviewed, pentested and audited regularly by us and 3rd parties so its not slop either. You are definitely leaving money on the table. We do mostly use chinese models on our own hardware (we colocate cages of racks) so this is not about Anthropic but about AI in general.

Skill athrophy is a real thing though; we try to prevent this by have hackethons (for lack of a better word) without AI where I pick something extremely non trivial and we implement it for fun and profit without AI (with would not matter much as they are currently bad at these things); last one was flex paxos for our in house db with obvious metrics for the endresult: data integrity (duh) under failure and performance better or at least the same as our raft production version.

good luck actually enforcing that.

> They're actively trying to use lobbying power to make open weight models illegal.

What is your evidence?

I've been plugging this perhaps too many times now, but I am trying to bootstrap a user-sourced corpus of exactly "what model is good at task X". So, not benchmarks, but high-level tasks. There's a bit of a ordering problem in that nobody wants to bother commenting on a site that has few comments - so PTAL and contribute if you can. https://model.reviews

I always use Opus 4.8 at max effort for everything. The $20 subscription didn't have enough tokens, but the $100 one had too many of them. So now I just max out Opus in order to maintain 100% weekly utilization.

Same boat as you, and my answer is "... Except when I ask and overall or checkup task that is specifically heavy or overseeing in which case I use the maximum level" which lately meant ultracode.

I'm not going to play around with thinking level every request because the goal is to make me save time not spend it in a different setting menu.

I tend to run it on High and then step it up for problems where I'm noticing it struggles, bump it back down after. Sometimes I accidentally leave a session in Ultracode for a day and wonder why things are taking so long, but generally happy with the results.

What I want is a harness that knows how to optimize this kind of thing for me.

It's really not that much. It's a bit hard to make sense of it not because it's hard to keep track of, but because they are being deceptive and opaque about what you're actually buying, and the thing you're paying for is different from one day to the next, as they fuck around with the parameters to boost subjective performance during a launch, then quietly degrade the service to cut costs.

Exactly this is my problem with all AI tools. I want someone else to create working tools for me so I can focus on my product. It is the same with other tools. I do not want to spent huge amounts of energy and time to setup my IDE, operating system or desk layout. I guess it is too early to have that now.

I also ended up using max effort/reasoning for both coding and general chat. They don't spend too much extra time on simple tasks these days.

Same advice as ever? We call it context engineering now, but prompt engineering still matters a lot. Most of the failures I run into are unspecified assumptions made by the model that derails the conversation, but usually updating the first prompt fixes it. Opus in my experience is a bit better about checking assumptions, while Sonnet will plow on ahead. An example is mentioning a file that doesn't exist: Sonnet will go ahead and try to grep your entire hard drive for it. Opus will say it's not local and request the path.

I trust neither for general knowledge and I still find Opus giving me answers that are completely BS. But the token spend for Q&A is nothing compared to coding, so I always use Opus + a lot of thinking. For coding, I find Opus to be better value/token but I haven't done any sort of rigorous test.

There are token optimization consultants that can help organizations find the right balance of models for their employees to minimize costs.

Just because it’s hard to keep track of doesn’t mean it’s not relevant.

Playing around with learning the differences is incredibly helpful to schedule on ones calendar weekly for an hour or two, while saving links throughout the week to try out.

It's almost like you want an automatically intelligent choice of your artificial intelligence.

Understandable frankly.

Just use deepswe as a reference point.

> This is why there was a "Sonnet only" usage bar for Max tier for the longest time.

it's still there. I still don't totally grok why I can't use all my tokens on Sonnet if I want to... maybe that signals something?

From my benchmarks, sadly, it doesn't seem to be the case much. Surprisingly. I found Sonnet comparable in speed to Opus (sic), but perhaps I was testing it wrong?

What's interesting, is that Sonnet 5 is actually worse[0] than 4.6 without reasoning.

It makes some sense, as models are trained more and more with reasoning, than without.

[0]: https://aibenchy.com/compare/anthropic-claude-sonnet-4-6-non...

You're the second person that has said this but I cannot understand why you are interpreting the "Agentic computer use" graph in this manner.

The graph shows that Opus is cheaper than Sonnet for the same performance. Unless I am suffering a cognitive blindness thing right now.

> Too expensive to perform daily tasks - open souce models are much cheaper

There is a real advantage, especially for businesses, in using an off the shelf solution from a corporate provider.

Personally, the advantage of not having to set up multiple solutions from multiple sources outweighs the cost of a $20 a month subscription. Think about why a lot of consumers prefer Apple devices over Linux. There are a lot of advantages to Linux, but "never having to think about my tools" is its own advantage.

The specific market positioning is... for me to use at my big tech company job, where we aren't allowed to use GLM and similar, but have fixed caps on how much token usage we're allowed to rack up a month.

No it doesn't? It's worse than Opus across the whole shared frontier on both plots.

They've absolutely both changed. The initial version I saw didn't include max effort data points on the first chart, and the plot itself was much less favorable to Sonnet at high/xhigh relative to Opus, but the new chart shows them as closer competitors. Weird.

I’ve found input tokens is around 5x more than output, so a task could be a couple million thinking tokens and then a few couple 100k output tokens?

3.5 Pro not yet launched, you mean 3.1 pro?

"but you can't trust 4.8 to go off on its own."

And what (avaiable) model do you trust to go off on its own?