This is what I've been wondering. I mean we all know they are compute constrained so I get that intuition to a point.

However, how would less compute result in degraded performance as opposed to more latency? Like is amount of compute something you can actually tune to control for latency? Wouldn't that imply they are actually serving requests in other models? If I run a very large model locally on a shit computer vs. something maxed out, it's the latency that differs.

The major providers like Azure along with OpenAI and Anthropic have the concept of "priority processing" which comes with service level guarantees on latency and token throughput. It's more expensive.

They basically dynamically allocate compute into this pool from the general pool to meet demand and SLAs. If demand for priority processing spikes latency in the general pool spikes.

Almost all subscriptions use general pool. That's why it gets really slow sometimes. Unless you use "fast" which acts like priority processing and is super expensive.

So that's part of how they balance compute based on demand and fund the "excess" capacity. This is a bit reductive and basic, but it's all the usual Cloud on demand, reserved, and spot capacity techniques at play.