| ▲ | askonomm 2 hours ago |
| That's an overly optimistic view of things. For a lot of us GitHub is critical infrastructure, which if it goes down loses us and our customers money. |
|
| ▲ | datsci_est_2015 2 hours ago | parent | next [-] |
| It is interesting, how much money is being lost during this outage? My significant other was just let go from their job as a scapegoat for an organizational error: 3 layers of failure - IC, manager, director, and the IC was let go. The error caused a 7 figure loss for the company that has 10 figures of revenue per year. The manager and director may not see any consequences, though the director will probably be forced out by end-of-year due to incompetence. The new executive has taken to firing employees much more eagerly than their predecessor, like some sort of Jack Welch acolyte. Their firing has put a lot of things into perspective for me. Mostly, fuck "at-will" employment and its negative effect on the American social contract. But also this "angry ranting" online that the original poster was referencing. Not everyone has the privilege to calmly respond to things that directly impact their livelihood. |
| |
| ▲ | theappsecguy an hour ago | parent [-] | | Any engineered system where an individual can accidentally cause a 7 figure outage is poorly designed. And engineering leadership that decider to terminate an individual due to such failure (as long as there was no malicious actions) is completely clueless. | | |
|
|
| ▲ | Waterluvian 2 hours ago | parent | prev | next [-] |
| I'd be curious what the statistics might actually be for people who are directly affected because their business is suffering vs. people affected because their employer's business is suffering. |
| |
| ▲ | 6LLvveMx2koXfwn 2 hours ago | parent | next [-] | | Most people's employment prospects are directly correlated with their employers ability to make money. | | |
| ▲ | john_strinlai 2 hours ago | parent [-] | | most people's employers arent firing people over a few hours of github outage. not to mention that any business which could potentially lose enough money that they would need to let go of developers from a github outage should probably already have some business continuity plans in place. |
| |
| ▲ | polycaster 2 hours ago | parent | prev [-] | | For Germany it's roughly 1:11 if you go by the self-employment rate (~8% of the workforce). |
|
|
| ▲ | nullify88 an hour ago | parent | prev | next [-] |
| GitHub Enterprise Cloud has been chugging along with no issues. I hope your critical infrastructure isn't dependent on a free tier / service. And that you have a business continuity process in place. |
| |
|
| ▲ | antaviana 2 hours ago | parent | prev | next [-] |
| "It's just money. It's made up. Pieces of paper with pictures on it so we don't have to kill each other just to get something to eat". Jeremy Irons in movie Margin Call |
| |
|
| ▲ | parthdesai 2 hours ago | parent | prev | next [-] |
| How is github being down losing you guys money? |
| |
| ▲ | dewey 2 hours ago | parent | next [-] | | If you pay developers x money / day and one of their core tools is down for n hours during the day and they spend their money on HN instead that's pretty straight forward to calculate. | | | |
| ▲ | aurareturn 2 hours ago | parent | prev | next [-] | | I was in the middle of a hot fix. Our pipeline goes through Github. | | |
| ▲ | parthdesai 2 hours ago | parent [-] | | That's understandable, but surely there's a way to bypass it? You need to have a breakglass procedures lol | | |
| ▲ | penultimatename an hour ago | parent [-] | | “lol just use a workaround” doesn’t work in an environment with hundreds or thousands of employees coupled with audit, security, and other legal requirements to ship software. | | |
| ▲ | parthdesai 44 minutes ago | parent [-] | | If you're actually bleeding money, you better believe you'll get permissions for a workaround, if you know what you're doing. It all ties back to the OP, where the issue you've might not be as bad as you think. I have been in situations where we have dropped all procedures to push a hot fix because we were actively bleeding money, and in situations where you know there is an issue, and you let it be. |
|
|
| |
| ▲ | phoronixrly 2 hours ago | parent | prev [-] | | Imagine Github being critical infrastructure for you... The ineptitude... | | |
| ▲ | parthdesai 2 hours ago | parent [-] | | We use github quite a bit as well. Github being down currently is not actively losing us money, i.e. having customer impact. |
|
|
|
| ▲ | kakacik 2 hours ago | parent | prev | next [-] |
| If its so critical why relying on it, and not having ie some mirror or some other way to handle any sort of outage like this. its not like Microsoft is your friend or good business partner, ever. With every single of these enterprise 'cloud' offerings you are giving (almost) complete power over your business/project to somebody else who couldn't care less about your success or failure, you are simply irrelevant for them. I see it at work too, every time critical external systems go down whole bank stops still, just because few bucks were saved yearly on some on-prem servers. Look at it this way, you are learning some important lesson today and finding great area of improvement for resiliency from now on. |
|
| ▲ | AlexandrB 2 hours ago | parent | prev | next [-] |
| Boomer opinion but trusting third parties to be critical infrastructure, especially with no SLA in sight, will always end in tears. "The cloud" is very convenient, but its providers will never care about your infrastructure or your customers as much as you will. |
| |
| ▲ | Maxion 2 hours ago | parent | next [-] | | Older millennial here and I agree. If you rely on Github you should at least be have your processes such that you can work around it. | | | |
| ▲ | otterley 2 hours ago | parent | prev | next [-] | | 3P-maintained infrastructure is what makes civilizations work efficiently. We're not all digging our own wells, generating our own electricity, and burning or burying our own garbage. | | |
| ▲ | kalaksi 2 hours ago | parent [-] | | You are still supposed to be prepared for outages. | | |
| ▲ | otterley an hour ago | parent [-] | | Agreed. So is the issue really then that people were inadequately prepared with backup plans and now they're suffering the consequences? It's not all that different from, say, an AWS region having a service impact. People would rather complain about AWS than prepare and utilize a well-tested recovery plan to shift to a standby region. Oftentimes there's no fallback plan because the business already considered it and decided it was too costly relative to the benefit, but when the incident happens, they still can't help but complain. Humans being humans. |
|
| |
| ▲ | aenis 2 hours ago | parent | prev | next [-] | | Boomer here as well, but I'd add that trusting your own org for critical infra usually also ends in tears. Most everything in IT involves failure, including in well designed systems designed by great engineers. I worked for a few years in an exceedingly well capitalised place which ran everything in their own data centers, money no object, with a truck parked somewhere, ready to go, with a smaller version of our critical infra. We had a serious business-stopping outage once every 18 months or so, every time for fringe reasons one only learns about when trying to run a large data center. Its convenient to blame the cloud and pretend that self-hosting in private sector was so, so great with six nines. | | |
| ▲ | 6031769 2 hours ago | parent [-] | | At this point github is barely managing one nine. | | |
| ▲ | aenis 2 hours ago | parent [-] | | True. I'd be very annoyed if I had anything mission critical on github. While we have everything in the cloud, we do actually self-host that part. |
|
| |
| ▲ | swills an hour ago | parent | prev | next [-] | | Agree. But also, it's affecting everyone equally, whether they have a free personal account or are part of an enterprise account with SLA. Understanding the practical value of an SLA is an interesting problem. | |
| ▲ | raffael_de 2 hours ago | parent | prev | next [-] | | +1 (as a millenial) ... especially given that setting up a git server for non-OSS company code isn't too much of a challenge really. also, no need to self-denigrate this reasonable opinion in preemptive obedience. | | |
| ▲ | otterley an hour ago | parent [-] | | GitHub is way more than just a git repo host. It manages code reviews, merge (pull) requests, and has an entire CI/CD workflow engine in it. Replicating all that is a challenge that most orgs are not up to. |
| |
| ▲ | poisonborz 2 hours ago | parent | prev | next [-] | | How did this become a boomer opinion? It is proved truth thousand times a day. Not that you shouldn't use third parties - but in this industry you can shrink this exposure to the minimum, and have plan B for anything else. | |
| ▲ | ethagnawl 2 hours ago | parent | prev | next [-] | | You're right. However, I'm also of the boomer opinion that you should get what you pay for. "Ranting online" about a service (you pay for) being unavailable is a reasonable reaction. It's not like they have a call center you can dial into for support ... | |
| ▲ | SideburnsOfDoom 2 hours ago | parent | prev | next [-] | | People are already saying things like "We need a plan B in case we urgently need to deploy a fix to production, and GitHub Actions is unavailable again". But in general, it's not feasible to do everything in house. And I don't think that GitHub is devoid of SLA: https://github.com/customer-terms/github-online-services-sla The issue is that they're not achieving two nines uptime in practice. | |
| ▲ | TZubiri 2 hours ago | parent | prev | next [-] | | Boomer who has never had a business? Depending on third party suppliers is business as usual. | | | |
| ▲ | BryanBeshore 2 hours ago | parent | prev [-] | | Truly a boomer opinion. Respect! |
|
|
| ▲ | TZubiri 2 hours ago | parent | prev [-] |
| And it's not just the user's fault, GH spent a considerable effort in marketing to position themselves as such, see Github Actions and similar junk. |
