The major providers like Azure along with OpenAI and Anthropic have the concept of "priority processing" which comes with service level guarantees on latency and token throughput. It's more expensive.

They basically dynamically allocate compute into this pool from the general pool to meet demand and SLAs. If demand for priority processing spikes latency in the general pool spikes.

Almost all subscriptions use general pool. That's why it gets really slow sometimes. Unless you use "fast" which acts like priority processing and is super expensive.

So that's part of how they balance compute based on demand and fund the "excess" capacity. This is a bit reductive and basic, but it's all the usual Cloud on demand, reserved, and spot capacity techniques at play.

▲

nonethewiser 16 hours ago | parent [-]

Thank you for the information but I'm not sure you understand my question. I'm not surprise to hear these details on how they control for latency.

Im asking on how lower compute could translate to worse model performance. That's the implicit claim being made when people are saying Anthropic nerfs a model (result is bad, not necessarily slow). The only way I can think that could happen is if they actually swapped Opus 5.5 out for a worse model that takes less resources to do inference.

	▲	Rapzid 15 hours ago \| parent [-]
		Yes, that's what they are claiming; that the full(er) precision models get heavily quantized and/or distilled to lesser, "nerfed" models and requests get routed to them based on load, time since release, time to new model release, and reading chicken bones. There is no proof, the frontier labs deny it, and certainly it's not happening on hosts like Azure, Amazon, and etc.. The claim is that it's "trivial" to detect observation, so they serve the normal model if you try to measure it.. Basically the conspricy theory is: * OpenAI/Anthropic nerf their models regularly to reduce compute demand * It's impossible to measure and thus prove