I usually tell people you don't need as much reliability as you think.

Three nines reliability is great for most purposes. 8 hours downtime a year.

If your system produces money at a constant rate, it captures 99.9% of the available money. Even two nines or one nine might be pretty good on that basis, when the alternative is spending 2x or 10x as much - let's build another unreliable system with that money that captures some other independent market opportunity.

Poor reliability is a problem where you need to chain many systems together, or where the cost of a single failure is very large compared to a success. Or - as happens commonly because of load - if your periods of unreliability are correlated with periods of maximum opportunity, like an e-commerce site failing on Black Friday or a trading system failing when the market is most busy. But if you don't have one of those cases, evaluate whether investing in reliability is actually worth it to you.

GitHub is an example where two nines of reliability ought to be OK. The argument against it is that it's bad marketing to have an unreliable service, especially one aimed at software engineers. And if GitHub is largely a marketing play by Microsoft anyway (do they really make back its cost in enterprise subscriptions?) then marketing considerations need to drive its reliability.

▲

YawningAngel 4 hours ago | parent [-]

I'm not sure that having your expensive software engineers not work 2.4 days a year is very palatable

▲

dmurray an hour ago | parent | next [-]

They already don't work for 130+ days of holidays, weekends, vacations and sick leave. What's another two days?

Probably I already ask them to spend 2+ days attending HR or compliance training or listening to senior management tell them about sales targets.

But the point is, that 1% extra productivity requires the sometimes staggering cost of making the software 10x or 100x more reliable.

▲

Dylan16807 3 hours ago | parent | prev | next [-]

They have other stuff to do that's not bottlenecked on push/pull/CI. 1% github downtime should reduce productivity by a lot less than 1%.

▲

himata4113 3 hours ago | parent | prev | next [-]

There are n services and n tasks, the chances of overlap are minimal. In most cases people take a much needed break during downtime.

	▲	dmurray an hour ago \| parent [-]
		Actually I think it's better if everything fails at once and everyone can take the day off (thanks, AWS!). Having your CI fail one day and your package repository the next might well cost you two days of productivity.

▲

vel0city 2 hours ago | parent | prev [-]

I fully agree with this take. We need to ensure we get every hour of work from our expensive engineers. This is why we got rid of coffee machines and bathrooms and moved to intravenous caffeine and other fluid drips and catheters. We cannot afford to lose productivity.