I find this hard to believe in this context. They should be utilizing load shedding or admission control and killing/rejecting jobs rather than hard failures if it is a scaling issue. It's much more likely an actual software defect than just more load. If this was the case (which they would likely prefer) free/public tiers would be removed first to preserve paying customers services.

▲

woodruffw an hour ago | parent [-]

Why not both? Higher base load combined with insufficient internal controls for ratelimiting/load-shedding (as in, they don’t know who to shed) would be explanatory.

▲

mgh95 an hour ago | parent [-]

If they can't implement something as simple as "decode upstream headers and determine if 429/503" I don't know what to say. Since this has knocked out all customers it indicates they likely don't have anything of this form implemented.

▲

woodruffw 35 minutes ago | parent [-]

I meant shedding of legitimate base load, not retries. I think we can safely assume they do the latter.

	▲	mgh95 4 minutes ago \| parent [-]
		[dead]