Personally, I've found myself getting lazier recently, trying to get a single session to handle tasks that should be broken into multiple sessions, etc. I've suspected claims of nerfing might come from changes in user behavior. I'm glad more disciplined checks are being run.

If I read your results correct it solved 5% more problems and used up between 3-5% fewer tokens, with a faster response time?

The spending limit definitely goes down much faster as of last week or so.

The problem is that it seems to be an A/B testing thing:

https://news.ycombinator.com/item?id=50422951

So only if you were "selected" for their experiments, your account was nerfed.

"Nerf tracking" is adversarial and exploratory. It's not something Anthropic wants anyone to do, and it's not something that the "nerf tracking" designers are necessarily doing in a foolproof way. Don't assume too much from them either way.

These nerf trackers do not work. I wish people would stop posting them.

They don't detect a quantized model until the quantization gets down to 4 bits or less.

Humans absolutely do detect quantization well before 4 bits...

Try it yourself if you want. Try a small local model, both quantized and not, and I would be very surprised if it isn't immediately obvious which is which.

Its incredibly easy to roll out a compute instance to X% of users, or route traffic between compute instances

so not everyone will have the same experience. If I was saving on costs that would be the easiest way to do it without deploying, without any nefarious intent. The persons in charge of that likely don't even know about how individual users are impacted

Nothing stopping Anthropic (or other labs) from pulling a VW Dieselgate and ensuring that benchmarks are routed to a non-nerfed model.

That must be on anthropic's benchmaxxing CI test

I hear a lot of accusations of nerfing as in, saying that the model itself has been quantized.

I doubt this is the case. I suspect all variation in quality and output can be explained in terms of prompt differences, and changing how much thinking happens at each level, as we see in the post above.

YES. To the point that I literally stopped using it, and have moved BACK to Claud.

This is getting ridiculous. The number of times I have switched from one model back to the other is crazy. I do not have any proof, so its quite possible I'm just delusional. AI psychosis? Either way, it FEELS like these models work really well for a while and then fall off. Switching to the latest model on the competitor always seems better, until it too seems to taper off. By then, a new model is usually available at the competitor, so I switch.

Its possible there's no degradation at all and just a PERCEPTION of it. But to me the end user, it really doesn't matter.

Both Astra and Opus didn't really looked good for the last week or two.

Yes I agree, Astra has been awful this week.

why were you skeptical of other users?

Sol been really bad in some basic stuff that I would remove it from frontier models entirely

GPT Astra felt nerfed just days after it launched.

I've been working with Opus 5.5 every day, probably around 10h on average, and I feel confident in saying that they did not replace anything.

I'm a bit disappointed with how this nonsense is spread across HN today.

I believe fraud is a sufficient term. You claim to be selling X but you actually sold them Y. Bait and switch is a selling tactic to avoid fraud where they make you believe product is X and then when you go to buy they tell you X isnt available but they have Y. While also unethical, it is not fraud because they did tell you what it is you are buying before you buy it.

if they just straight up dont sell you something you bought then its fraud.

I think there is a ripe opportunity for expanding the definition of fraud to include changing the "ingredients" of a named product on the market. It is not limited here, we see it with food, and even in the medical field

(I will die on the hill that medical fraud was committed when mRNA therapy was used as a treatment for covid and it was 'sold' to the public as a "vaccine" despite not being the same thing as any vaccine before it. The definition of vaccine was expanded to include it without sifficient public disclosure to warrant informed consent imo. This is not a statement about conspiracy or the efficacy of vaccines or a moral judgement on whether it was for the greater good or not.. it was simply an act of fraud that robbed the public of proper informed consent by making people believe it was X when it was really Y.)

So yes, I think there is a large need for the public to at the very least put a strong handle on the ability to change what a product name means. It is a trivial problem to solve considering it is only a problem due fraud or fraud adjacent activities.

heres the solution: You have your product name, and then any changes to it REQUIRE a new version name. Like how you have Coke and Diet Coke, only formalized such that Coke cannot legally replace the ingredients of regular Coke with the ingredients of Diet coke and still call it Coke. With Software they could at least use version numbers and release modifiers. most of the time its built in already, but it needs to include the full stack. if they change anything that has an interaction with the user it should change the label seen by the user for the product.

thank you for coming to my ted talk

They do change things though which can have impact on behaviour.

System prompt would be the easiest one.

But there were flash or inbetween models, different checkpoints etc. THey might also try different hardware which might not be binary compatible.

Interesting - I had the same feeling but mostly with 4.6 when I "downgraded" for some easier/trivial tasks: The model forget things I posted 20 min ago in the chat (and the same file was in the context as well)

“Why now” - they are about to IPO. It has been obvious for months that they would drop a new flagship model at exactly this point.

It would be funny though if you are right and that model is Fable 5.5 in a couple weeks.

Do you know of any rumors if they are still planning to release Fable 5.5 before the IPO or today's 5.2 is it?

Dario is gonna have to honeypot someone from Alibaba to get ahead.