I at least didn't see this. I had a baseline evaluation from before Nov 3 on Opus 5.5 medium, so I re-ran it this morning.

Eval context: 30 bug fixes and features evaluated by tests the agent didn't see. Ran on Opus 5.5 at medium effort in Claude Code.

20 of the tasks were last run on Nov 1, twice each. Nov 1 solved 36/40, and today solved 19/20. Thinking tokens per task were 0.95x Nov 1’s and output tokens 0.97x. The one significant change was speed: median time to first token decreased from 3.0s to 1.7s.

Caveats: one run per task today, medium effort only.

▲

marshalla 2 hours ago | parent | next [-]

Personally, I've found myself getting lazier recently, trying to get a single session to handle tasks that should be broken into multiple sessions, etc. I've suspected claims of nerfing might come from changes in user behavior. I'm glad more disciplined checks are being run.

	▲	mr_roboto an hour ago \| parent [-]
		It could be harness changes too, been quite unhappy with opus 5.5 the last few days and there have been a lot of harness updates.

▲

janalsncm 12 minutes ago | parent | prev | next [-]

This is an example of LLM text that doesn’t make sense.

Paragraph 1 says there were 30 bug fixes.

Paragraph 2 first says they tested it on 20 of the “tasks”. But why is the denominator 40? Where did that come from? It should be 30.

It goes on to mention another test, this time with 20 tasks. Again, it’s unclear where this is even coming from. Based on the first paragraph it should be 30 not 20.

▲

drakythe 3 hours ago | parent | prev | next [-]

If I read your results correct it solved 5% more problems and used up between 3-5% fewer tokens, with a faster response time?

▲

ariwilson 3 hours ago | parent | next [-]

Correct. But the 5% more solves and 3-5% fewer tokens are within the noise of previous eval runs I've done on models. The faster response time is not.

▲

Spivak 3 hours ago | parent [-]

Someone using an actual statistical test to quantify if there's actual measurable improvement?! An angel among men.

▲

jerkstate 2 hours ago | parent [-]

from my basic stats knowledge, it's not reliable to calculate the confidence interval of the sample proportion with fewer than about 10 samples on both sides. So if you're only seeing 1 or 2 failures in 20, and that approximately held as you scaled, you would have to run about 200 tests with each in order to get at least 10 failures in order to calculate the confidence intervals of the two samples and see if they overlap or not.

	▲	disgruntledphd2 2 hours ago \| parent [-]
		And this is why not many people do this. Even if you have the decals and the statistical knowledge, it's pretty expensive to do this regularly.

▲

astro1234 2 hours ago | parent | prev [-]

Uncertainty from count statistics are far larger than this. These are two perfectly consistent results there’s no statsig difference

▲

risyachka 2 hours ago | parent | prev | next [-]

The spending limit definitely goes down much faster as of last week or so.

▲

2 hours ago | parent | prev [-]

[deleted]