| ▲ | BryceEller 2 hours ago |
| I dunno there are some other nerf trackers that aren't showing a regression: https://github.com/ninjahawk/livenerf |
|
| ▲ | xtracto an hour ago | parent | next [-] |
| The problem is that it seems to be an A/B testing thing: https://news.ycombinator.com/item?id=50422951 So only if you were "selected" for their experiments, your account was nerfed. |
| |
| ▲ | Tadpole9181 31 minutes ago | parent [-] | | But the claim here is "the models are nerfed before a release", not "they A/B test quants / efforts". If it were A/B testing, it doesn't really make sense to correlate it to release windows at all. |
|
|
| ▲ | swatcoder 2 hours ago | parent | prev | next [-] |
| "Nerf tracking" is adversarial and exploratory. It's not something Anthropic wants anyone to do, and it's not something that the "nerf tracking" designers are necessarily doing in a foolproof way. Don't assume too much from them either way. |
| |
|
| ▲ | rybosworld an hour ago | parent | prev | next [-] |
| These nerf trackers do not work. I wish people would stop posting them. They don't detect a quantized model until the quantization gets down to 4 bits or less. Humans absolutely do detect quantization well before 4 bits... Try it yourself if you want. Try a small local model, both quantized and not, and I would be very surprised if it isn't immediately obvious which is which. |
| |
|
| ▲ | yieldcrv 2 hours ago | parent | prev | next [-] |
| Its incredibly easy to roll out a compute instance to X% of users, or route traffic between compute instances so not everyone will have the same experience. If I was saving on costs that would be the easiest way to do it without deploying, without any nefarious intent. The persons in charge of that likely don't even know about how individual users are impacted |
| |
| ▲ | olalonde an hour ago | parent | next [-] | | At what point does this become false advertising, or even fraud? Imagine paying AWS for a high-performance server, but 5% of customers who order one quietly get a slower, cheaper server and still pay the full price. That's a serious claim and makes me skeptical Anthropic is doing something like that without more evidence. | | |
| ▲ | kuboble an hour ago | parent [-] | | Actually, as a person who rented a dedicated server or two (not from aws) this exact thing happens quite often. For example a provider quietly underclocks your cpu or puts it is an energy saving plan hoping you won't notice. |
| |
| ▲ | cft 2 hours ago | parent | prev [-] | | The degradation only happens to some accounts but not to all. https://www.reddit.com/r/ClaudeCode/s/64DXRERLyU It would be nice to know how they select. I max out my $200 subscription every week and I was selected. On Saturday, the Opus on my account that happened to be supervising a running live process, became unusable. | | |
| ▲ | tmpz22 2 hours ago | parent [-] | | It would be trivial for them to opt-out accounts that are doing benchmarking from a planned regression. I really really really do not trust companies on the verge of IPOing from goosing their numbers. Even if the lawyers are up their ass, the social pressures are more than enough to compel the behavior. For every good person willing to do the right thing even at the cost of a delayed career progression, there are 20 people willing to do anything not even for a dollar but for an "atta boy!" from the boss. | | |
| ▲ | cft 2 hours ago | parent [-] | | That's a major reason to use local models. Or open weights models on a trusted cloud provider. Even if they are way less capable, they're more predictable |
|
|
|
|
| ▲ | scrlk 2 hours ago | parent | prev | next [-] |
| Nothing stopping Anthropic (or other labs) from pulling a VW Dieselgate and ensuring that benchmarks are routed to a non-nerfed model. |
|
| ▲ | fritzo 2 hours ago | parent | prev [-] |
| That must be on anthropic's benchmaxxing CI test |
| |
| ▲ | chinathrow 2 hours ago | parent [-] | | Nerfmaxxing. It probably need a whistleblower to uncover what they are doing. |
|
