The August 17 outage, and the work ahead

▲ The August 17 outage, and the work ahead(github.blog)

17 points by 0xedb 25 minutes ago | 12 comments

▲ blakesterz 6 minutes ago | parent | next [-]

  "Since April, monthly commits have grown from 1.4 billion to 2.9 billion. "

Wow, that is some incredible growth in a really short time.

▲ ivraatiems 8 minutes ago | parent | prev | next [-]

"We are committed to fixing these problems, as long as it doesn't involve buying things other than AI computers, hiring humans, or using non-Microsoft products."

Calling Azure the solution to this problem when it is in fact the source of most of these problems is just fantastic doublespeak.

Github is ripe for disruption and I hope it is disrupted soon.

▲

dcrazy 5 minutes ago | parent | next [-]

> We installed as much hardware as available power allowed in our existing data centers while accelerating our migration to Azure.

And from the RCA [1]:

> The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic.

[1]: https://www.githubstatus.com/incidents/zkxwbgr0cnmx

▲

mort96 5 minutes ago | parent | prev [-]

If you're a big company, you can afford having one engineer spend one or two days per year to maintain your self-hosted GitLab or Forgejo. On top of better reliability than GitHub, you'll get the additional bonus that your source code won't accidentally leak through being in Copilot's training set.

If you're a hobbyist, Codeberg is great, has a nice community and automatically shields you from slop contributions.

	▲	ivraatiems 3 minutes ago \| parent [-]
		The issue with these systems is that they lack Github's sophistication for issue tracking, knowledge transfer, and automation. I think Gitlab is a mature product in its own space and unlikey to change, for instance, at this point. Codeberg also has the issue of having a political stance which means they will not accept just anyone's use of the platform. That is absolutely their right and I have no issue with it, but it's unattractive to me - as someone who agrees with most of their current politics - because the day they decide they don't like me, I'm screwed.

▲ jdm2212 5 minutes ago | parent | prev | next [-]

> Errors in those services triggered a client-side retry loop that increased traffic during recovery.

The worst outages I've been part of always have some version of this :(

▲ lenerdenator 2 minutes ago | parent | prev | next [-]

We need to have a package of FLOSsoftware that you could run on the cloud of your choice that offers most of what GitHub does (niceties on top of Git) without the centralization.

GitLab was close last I remember but there was some sort of enterprise tier when I tried hosting stuff on a local server years ago. I want true FLOSS, not another SaaS equivalent of the coke dealer giving clients the good uncut stuff when they're just starting out only to sell crap when they're addicted.

▲ monlockandkey 2 minutes ago | parent | prev | next [-]

They should rewrite their Ruby code to a performant language.

▲ amazingamazing 4 minutes ago | parent | prev | next [-]

Exponential growth. No company could handle that without some issues. Good luck to them. And for those who cannot tolerate this, there are many self hosted options.

▲ rvz 3 minutes ago | parent | prev | next [-]

And another outage. [0] Looking forward to the subsequent post-mortem on that one.

You might want to not go all in on GitHub anymore since it is very unstable to use. A self-hosted instance would have a far better uptime than GitHub over the years.

6 years ahead [1] on not going all in an centralizing everything on GitHub.

[0] https://www.githubstatus.com/incidents/bhbcjn4n3jzp

[1] https://news.ycombinator.com/item?id=22867803

▲ nycpig 4 minutes ago | parent | prev | next [-]

[dead]

▲ codegeek 14 minutes ago | parent | prev [-]

[dead]