Remix.run Logo
Waterluvian a day ago

I have no hard data but I have a strong feeling this morning that something's wrong with Fable 5 compared to Friday evening.

Just an hour ago I had Fable correctly identify an unused method that could be deleted. I then immediately get a diff for an exact duplicate method, and then Fable outputting, "I accidentally duplicated <method> instead of deleting it. Removing both copies now."

The remaining morning complaints that makes it feel like something's off is that it will do a lot of "thinking" for simple things that previously took very little time. And it got very lost and completely mixed up DE-91M predicate names and implementations. Just absolute disaster code that I had over the past months come to generally expect it to do without issue.

Glad I carefully review everything. I think what I need is reliability and consistency. But it feels like picking a model from the list doesn't guarantee that: that the models' "brain" is open on the table and they're screwing with it.

fnordpiglet 21 hours ago | parent | next [-]

It’s load shedding. They’re reducing consumption for capacity balancing at your expense. Whenever there are rate limiting storms Claude gets dumber. They also shift capacity for new releases, and Claude gets dumber leading up to it.

Self run infrastructure won’t have this cost but you have to manage the capacity and rollouts yourself, at which point it’s more obvious what’s happening, but the effects will be the same. The not knowing makes it harder, but also harder to plan your own work around.

rfgplk 20 hours ago | parent [-]

Correct, but they should explicitly announce this ahead of time.

pixl97 20 hours ago | parent [-]

A general rule of corporate behavior unless they are forced to under duress.

If this is duress of competition or at gunpoint of regulators is up for the population to decide.

le-mark 15 hours ago | parent [-]

Luckily for us open weight models exist. Until regulatory capture anyway.

fidotron 21 hours ago | parent | prev | next [-]

The Claude models definitely felt more susceptible to moods, like you could leave them for a few hours, come back and it suddenly was unable to do things which it was doing just earlier, which tellingly is never an experience I've had with an open model.

Honestly I lost patience with Anthropic both clearly messing around with things like this and their agitation over regulation. They aren't good actors, and quite why so many blindly trust them with their company crown jewels is a mystery.

espeed 21 hours ago | parent [-]

Claude Code's prompt cache expires after 1 hour.

dwaltrip 20 hours ago | parent | next [-]

The cache shouldn't affect inference. It is purely an I/O optimization.

desterothx 18 hours ago | parent | next [-]

I think it should, as you dont need to use the encoder layer on the new tokens, you just read the embedding from the cache. that's why cache reads are cheaper

dwaltrip 17 hours ago | parent [-]

I meant, it shouldn't affect the resulting LLM output. It's a performance optimization that doesn't change the behavior.

cma 13 hours ago | parent | prev [-]

[dead]

namrog84 21 hours ago | parent | prev [-]

Is that from start of a new conversation per conversation?

espeed 20 hours ago | parent [-]

It's supposed to be for token optimization (https://code.claude.com/docs/en/prompt-caching), but are people experiencing degraded performance when you let Claude Code sit for hours/days and come back?

whalesalad 20 hours ago | parent [-]

yes, 100%.

prodigycorp a day ago | parent | prev | next [-]

New release of fable and opus 5.5 is pending and Anthropic is reallocating resources. Degradation always happens in transition, it sucks.

Opus 5.5 is being served under opus 5 right now.

gslepak 21 hours ago | parent | next [-]

> Opus 5.5 is being served under opus 5 right now.

On what basis are you claiming this?

prodigycorp 13 hours ago | parent [-]

They’ve been secret serving it. Try ask if they know who tibo the reset guy is. If they know the answer it’s the new version.

SequoiaHope 21 hours ago | parent | prev | next [-]

Can you elaborate on the mechanism of this degradation? If resources are not available I would expect a request to fail with a message about resources not available. Do they tweak back end model capabilities to maintain service in a degraded state?

arcanemachiner 21 hours ago | parent | next [-]

Dollars to donuts, they are speculating, and not privy to inside information on the topic.

However, I believe that runtime model quantization is possible with some publicly-available inference engines (e.g. vLLM), so its not beyond belief that the closed labs do quantize at runtime, either to allocate compute, or to nudge users towards a preferred model (e.g. make the incumbent model dumber to push people to use the latest-and-greatest model, or vice versa to ease the load on the latest model, which is typically larger than the old one).

sznio 5 hours ago | parent | prev | next [-]

I don't work at Anthropic, but I would assume they could serve smaller quantizations during peak hours - this effectively controls the "resolution" of the model. They could also control the resolution of the KV cache, which would make the model not necessarily dumber, but worse at understanding the incoming requests. And finally, you could pass off what was "high" effort as "extra", because why not.

rybosworld 20 hours ago | parent | prev [-]

An AI lab will never volunteer the information because it opens them up to lawsuits if they are purposely degrading service and not letting users know.

They can limit how hard the model thinks for a given effort. Suddenly xhigh only thinks as hard as high did, and high shifts down to medium effort, and so on.

They can also serve quantized models. And this has the benefit of practically not showing up in benchmarks at all even if the user experience is obviously degraded.

The other major thing the labs do is silently drop the usage limits. This has become very noticeable for codex users who are suddenly burning through their weekly usage in a few hours.

pixl97 20 hours ago | parent [-]

Yea, if you ever run your own models on a GPU there are a whole ton of different dials you can adjust that drastically affect compute use, memory use, and output token quality, and number of tokens held in memory.

If anyone reading has a GPU it's worthwhile just messing with a smaller model for a bit to watch how the settings affect output.

pllbnk 21 hours ago | parent | prev | next [-]

It shouldn’t be an excuse. They are selling a product and that product should always be within the quality range.

user43928 20 hours ago | parent [-]

And it's not. A conspiracy theory is what it is.

I have no reason to doubt the claims of the employees at OpenAI and Anthropic who have told us personally multiple times, including here on HN, that they do not degrade the models in order to reduce load.

As for the endlessly long analysis in the OP, it appears it's based on analyzing their random usage data rather than any fixed benchmark. I don't think it makes much sense.

w1296 a day ago | parent | prev | next [-]

Especially with the frequent releases aka version bumps.

21 hours ago | parent | prev | next [-]
[deleted]
pertymcpert 20 hours ago | parent | prev [-]

Why would reallocating resources make a single inference run worse in quality?

carljungslabtek 11 hours ago | parent [-]

Isn’t the idea that they’re limiting the amount of gpu time normal users get to spend on the “thinking” portion of their query?

I think that’s the claim in the post, that even though no one can see the true chain of thought, that even the “thinking” text that does get exposed to the user is shorter given the same prompts over time. Not saying it’s true but I think that’s the claim. I’ve personally never noticed the alleged “nerfing” with my enterprise use at work or my subscription use at home which is only during off hours.

zarmin 21 hours ago | parent | prev | next [-]

I would rather wait in a queue than be routed to a degraded model. And if they _have_ to degrade the models, then I wish they would fucking tell us. Instead, it's "I have a strong feeling".

That we have to guess at this is by far the worst part of the AI era. It feels like a dark cloud over my productivity. It makes my body tense for the entire day when it happens. Not healthy.

meowface 21 hours ago | parent [-]

They have repeatedly said they do not ever intentionally reduce model quality and do not degrade in this way, and that a model version number is always the same.

But, of course, OP is an empirical claim to the contrary, and I'd be curious to see if anyone (who's been capturing data over these timeframes) can replicate the same results and if Anthropic has any comment.

mh- 21 hours ago | parent | next [-]

Every official statement I've seen around this is careful to say that they "don't intentionally reduce model quality", which leaves plenty of room for "we adjusted some knobs and our evals show performance is materially the same".

However, I also agree that I haven't seen any robust data from someone tracking it daily/weekly. The handful of sites purporting to do this aren't even running it enough times to hit stat sig.

edit: someone linked one elsewhere in this thread called AI Stupid Level - they "run 7 trials instead of just 1". I don't blame them. Doing this in a statistically sound manner would cost a small fortune.

meowface 20 hours ago | parent [-]

I kind of feel "reduce the amount of thinking tokens produced" would fall under degrading model quality.

In any case, I am willing to believe it's possible something degraded, but so far I have not seen any empirical evidence of it since the previous incident with the inference and harness bugs. I lean towards Anthropic probably not intentionally doing anything like this without disclosing it beforehand.

pixl97 20 hours ago | parent | next [-]

The issue here is you have to think like a lawyer trying to weasel out of making an empirical statement.

For example "We didn't change any settings, but when GPU use gets high the run time of a prompt is lessened. But you must remember this is always in effect so nothing changed at all. This happens occasionally on random prompts some of the time, and when it's busy it happens all of the time".

In someones eye this would fit the letter of the law but not the spirit of the law that you hold.

20 hours ago | parent | prev [-]
[deleted]
cma 13 hours ago | parent | prev [-]

See March 26, 2026 incident. Model not degraded, but harness changed to strip out past thinking tokens when a session went out of cache to save money and ease capacity constraints (affected API users too), resulting in bad degradation.

bitlad 21 hours ago | parent | prev | next [-]

Sounds like me without coffee.

rfgplk 20 hours ago | parent | prev | next [-]

> I have no hard data but I have a strong feeling this morning that something's wrong with Fable 5 compared to Friday evening.

Fable is effectively worse than Opus 4.6 now. They severely messed with the model.

JMKH42 21 hours ago | parent | prev | next [-]

If you follow reddit forums for claude code, its common to see people, on the same day, claiming that Opus/Fable is especially smart today, and especially dumb today.

I think people are still not used to non deterministic tools like this, and human perception is absolutely horrible at evaluating trends like this no matter how smart, clever, and experienced you are.

If you have a bank of rigorously tested benchmarks that you run every few days, with enough trials to know what your standard deviation is, and you are getting significant trends over time with those, that would be interesting.

But "I have a feeling" and "Seems like" really isn't a reliable signal at all, humans just can't handle perceiving these things reliably. On top of that changes in your work environment can easily pollute LLMs and change quality of results. Are things getting added to your memory or claude.md files that you don't realize? Is your project growing in size and thus claude is performing worse as more context is needed to work with it? etc etc

sigbottle 20 hours ago | parent | next [-]

> I think people are still not used to non deterministic tools like this, and human perception is absolutely horrible at evaluating trends like this no matter how smart, clever, and experienced you are.

The implication is that humans are unreliable and shouldn't be trusted.

Or humans have certain shorthands when they complain on reddit, but their diagnoses are accurate for the specific context? If my AI does something stupid, am I not allowed to call it out? A NS-solving AI is still capable of not satisfying the abstract thing called the user experience. People have intelligent thoughts without compiling to lean.

OK, you say. Then let's get an aggregate benchmark for "intelligence". That doesn't prove that AI didn't flounder a specific use case that the user requested.

Classic moves: Humans are unreliable, converge to some "objective" benchmark that necessarily will quotient out the special cases, etc. Wonder how we'll be solving these issues in the AGI era - well, if you have an AGI that just replicates itself, dominates everybody because it's a machine and humans are soft fleshy creatures, and agrees with itself, fine. But part of the beauty of human experience is the messy part, and providing value is in the messy part.

nomel 20 hours ago | parent | prev | next [-]

> and human perception is absolutely horrible at evaluating trends like this

The need to have a mental measure of competence for your fellow man is, most likely, a pre-human skill, probably with a dedicated bit of neurons for it. I think the problem is that those instincts were co-evolved with our fellow man, and, as you say, don't apply at all to a more non-deterministic system that, fundamentally, lacks some logic faculties that even small children have (simple riddle modifications, car wash question, etc).

pixl97 20 hours ago | parent | prev [-]

In other industries of chance we have regulators that ensure compliance and that the providers aren't cheating.

At the end of the day the highest quality of benchmark tells you nothing if the man behind the curtain is constantly changing variables on you. You have no idea if you're really testing the same thing at all. So when you run your test at the top level on their system you're seeing lets say a 30% difference in quality most of the time, you have no idea if you should really only see a 5% difference in quality if you were running a local model with stable settings.

w1296 a day ago | parent | prev [-]

Maybe they are jealous of Navier Stokes and try the Hodge conjecture with 80% of total compute at the expense of their customers.