Remix.run Logo
PaulKeeble 4 hours ago

One thing that is I feel missed about uptime percentage when compared to on premise uptime is when the downtime occurs. Its far more impactful if its in the middle of the working day or during the busy period of shopping. A store that goes offline in the middle of black friday or in the run up to Christmas is harmed a lot more than some down time on a Sunday night/Monday morning at 3am.

One thing I have noted over time is a lot of these AWS, Azure et el downtimes is they occur in the middle of everyones day, millions of people are impacted by them. Same with github its getting in the way of work. Whereas when we hosted services on our own equipment the downtime was usually out of main hours. The percentages are in many ways the wrong measure of downtime because hours aren't equal in impact to businesses.

bryanlarsen 4 hours ago | parent [-]

It's fairly safe to assume that most outages occur at the worst time possible. They tend to occur at times of high load.

hinkley 36 minutes ago | parent [-]

Surely there's a word for this, but I haven't learned it.

The problem with load balancing also shows up in employees mass quitting. It's the same queuing theory problem but I don't know the name of it.

Essentially, when you run a set of 'resources' near capacity, if one fails then the next most highly loaded one also goes past capacity and also fails. The work keeps getting foisted onto other units that also fail.

In retail, the beleaguered employee quits in frustration, and the three other people who can do the same onerous tasks the quitter did now find their jobs have gotten 33% worse with no extra remuneration for the insult, and one of them quits too. The last one or two simply refuse to do the task more than they used to, and the boss can't threaten them with being fired or reported for it because if they quit then the business ends. And still the whole thing goes off the rails.

That's why consistent hashing divides the work of one failed node and spreads it evenly over the remaining nodes, so that a spike is softened as much as possible instead of being concentrated on one or two fallbacks who also immediately fail. It's not that it's a great solution, it's that it's the least bad option amongst some pretty terrible ones.

The solution is to undersubscribe the hardware, which some beancounters hate. But most bean counters don't understand Queuing Theory.