| ▲ | outworlder 3 hours ago |
| At my company, we migrated basically all CI/CD to Github Actions. Also our Gitops processes. I have warned that putting all the eggs in one basket and they key to the fridge in the same basket was a bad idea. People didn't care. So I don't care anymore. Github seems to have degraded measurably after they got acquired. As is tradition. |
|
| ▲ | nrmitchi 3 hours ago | parent | next [-] |
| > Github seems to have degraded measurably after they got acquired. As is tradition. I'm not disagreeing with you that it has degraded, however it was acquired almost 8 years ago at this point, and the software world has been turned on it's head in the last... 3? Software generally degrades post-acquisition, I'm not denying that. Github has definitely degraded. I'm not sure I would agree that the core reason in this case is "the acquisition" and not "AI making software and free-CI-load demand 3-5 orders-of-magnitude higher than it was 3 years ago" |
| |
| ▲ | JohnMakin 3 hours ago | parent | next [-] | | The latest problems correlate pretty closely when they tried to migrate the entire platform onto azure in an insane timeline, something like ~1 year (for a project of that size, that's borderline insane). | | |
| ▲ | doomroot13 2 hours ago | parent [-] | | Yes, but also in combination with the not-borderline-and-actually insane growth in commits and load on the platform due to AI. | | |
| ▲ | sleepybrett an hour ago | parent [-] | | this is often the argument that is trotted out, but i don't see any evidece of that. | | |
| ▲ | JohnMakin an hour ago | parent [-] | | It has been reported new user volume has nearly doubled, and commit volume 10x+ - that could certainly have an impact, but "Azure" and "scale" in my career are words that have never really gone together. Azure Cloud is the only 'hard no, will not interview' of any cloud provider that's listed on a job posting, and I've worked with just about all of them by now I think. Nothing even comes close to such a miserable experience. The thought of migrating something at the scale of github to azure gives me the cold sweats. |
|
|
| |
| ▲ | mday404 2 hours ago | parent | prev | next [-] | | Agreed. It's not at all clear that GitHub as an independent company would be doing much better at dealing with this new world than GitHub within Microsoft is doing. | | | |
| ▲ | nvme0n1p1 2 hours ago | parent | prev | next [-] | | > I'm not sure I would agree that the core reason in this case is "the acquisition" Would data change your mind? https://damrnelson.github.io/github-historical-uptime/ | | |
| ▲ | cstrahan an hour ago | parent | next [-] | | Playing devil's advocate: from the graph alone, it could just be that the status page became more accurate, so same amount of down time, but reported more now. On the other hand, I don't remember things being this bad before the acquisition. | |
| ▲ | nonethewiser an hour ago | parent | prev | next [-] | | That is quite stark. Whats the deal with the colors? Red and yellow don’t strictly seem to mean higher/lower. Some yellow dots are lower than red. | | |
| ▲ | zimpenfish an hour ago | parent [-] | | As best I can tell, the red dots are "major outages", the yellow are "partial outages". |
| |
| ▲ | an hour ago | parent | prev [-] | | [deleted] |
| |
| ▲ | shimman 3 hours ago | parent | prev [-] | | These are incredibly complex and massive systems, they do not fail instantly. The failures happen incrementally until they become insurmountable and the system collapse. Incompetence doesn't show its head immediately, it can take quite a while. Either way, it's clear the acquisition has not helped github in any meaningful way. GitHub is owned by a multi trillion dollar transnational corporation. They have enough money, resources, and human labor to achieve whatever they want, but this is what they want and this is what they've achieved. |
|
|
| ▲ | neya 3 hours ago | parent | prev | next [-] |
| It's Microsoft's touch. Anything they touch gets fucked up. It's because of their shitty infrastructure aka Azure. There was an employee here who actually gave insight into how bad the situation was a while back. Just reading that should make you re-evaluate anything owned by Microsoft. Edit: here's the link: https://isolveproblems.substack.com/p/how-microsoft-vaporize... |
| |
| ▲ | ebri 2 hours ago | parent [-] | | Would be very interested in reading that, do you have a link or a hint so i can find it? | | |
|
|
| ▲ | sandeepkd 3 hours ago | parent | prev | next [-] |
| I feel sorry for the teams at this point. These kind of incidents at this scare and frequency are bound to create an acrimonious environment where everyone would be looking for scapegoats on the pretext of finding root causes. |
| |
| ▲ | sleepybrett an hour ago | parent [-] | | it's almost like they should have a test environment with proper load testing... |
|
|
| ▲ | coldpie 3 hours ago | parent | prev | next [-] |
| > People didn't care. So I don't care anymore. Yeah. Enjoy the rest of your day off. Winter's coming, maybe go hit a park for a few hours or play a new video game or something. Or take some time to learn a new skill or something. Not your fault if the company can't provide the tools to do the job, go find something better to do. |
| |
| ▲ | hack1312 3 hours ago | parent | next [-] | | Great ideas if working remote, can’t imagine busting out my Steam library in the office though. | | |
| ▲ | coldpie 3 hours ago | parent | next [-] | | Time to bring back the Boss Button https://en.wikipedia.org/wiki/Boss_button :) | |
| ▲ | worble 3 hours ago | parent | prev | next [-] | | Then its time to break out the good old MUDs | | | |
| ▲ | zhengyi13 2 hours ago | parent | prev | next [-] | | True story: I was working in the Techstop (40) @ Google, 2011-ish, when a nice person from PeopleOps came through and politely requested my admin powers to help them install Steam on their corp-issued Windows laptop. I'm sorry, what? Well see, PeopleOps was going to have an onsite intending to help recruiters better understand Engineering types, and part of the whole business was going to involve everyone firing up a copy of Portal and playing a puzzle-solving video game. So I immediately got to it. Never heard back as to the success of the endeavor, but turns out there are circumstances where you can break out your Steam library at work. | |
| ▲ | sleepybrett an hour ago | parent | prev | next [-] | | we had a member of my team at a company about ~15years ago that they tried to fire for installing steam on their corporate laptop when they had sent him to another city for for what was supposed to be a week but ended up being three weeks. this was against our wishes (he didn't need to be there, there was no good reason for it, he could do all the work remotely from our office and had been doing it successfully for three months, the project was 'at risk' due to schedules but none of that was our fault at all). I had a knockdown drag out fight with both HR and Security. I agree that it was probably better for him to have packed his own personal laptop or similar gaming device but security had shown absolutely no risk and it was clear that HR was just 'clucking to cluck'. Eventually they relented with a mealy mouthed 'warning'. The fact that he had installed steam after his stay had been extended and they couldn't show any evidence that he was playing during working hours was a hillarious, and lucky, footnote. | |
| ▲ | esseph 3 hours ago | parent | prev [-] | | Coward! /s |
| |
| ▲ | 999900000999 3 hours ago | parent | prev | next [-] | | That’s a good way to get blamed and let go. But sometimes it’s a catch 22. I’ve had numerous times where systems simply didn’t work. I brought it to my managers attention and they didn’t care. Then later I leave in disgust or get fired. All my fault. I should have used my magical powers to convince my manager he’s wrong while not making him feel stupid. Ultimately if you’re on the titanic it’s easier to grab a life boat vs trying to avoid the ice berg. | |
| ▲ | JsonDemWitOster 3 hours ago | parent | prev [-] | | Isn't this basically https://xkcd.com/303/ but "Github is down"? (Man, unbelievable the mileage we're getting out of this comic.) I mean this really seriously. Maybe not a game on your computer, that's unprofessional (unless your TL/manager does it first). But > Not your fault if the company can't provide the tools to do the job, go find something better to do. is just Truth. What should I do in this case to best deliver value to the company, fly to SF and hand Github a ringbound volume of my changeset? Patch our corporate network so we can interface to Github via RFC 1149? More "professional" ideas: - read a book; a novel or TAOCP for nerdmaxxing - play a game on your phone - learn to draw. Draw hands! |
|
|
| ▲ | jbs789 3 hours ago | parent | prev | next [-] |
| Sounds like you now have your audience for the previously proposed solution. |
| |
| ▲ | dewey 3 hours ago | parent | next [-] | | The attention for ideas like that usually fades as quickly as it came after the downtime is over. | |
| ▲ | outworlder 3 hours ago | parent | prev [-] | | After we spent a couple of years ripping and replacing the previous solution? You may have a point. I guess I can embrace the corporate way, repackage the old solution with a few improvements as something brand new, and pitch to execs. |
|
|
| ▲ | gateauxd 3 hours ago | parent | prev | next [-] |
| Curious what even is the alternative. The stickiness is quite high and risk is scary to swap providers. |
| |
| ▲ | x86a 3 hours ago | parent | next [-] | | Claude/codex has made the stickiness much less of a problem. We just say "no new repos in GitHub" and then put agents on refactoring the existing repos for GitLab compatibility. | | |
| ▲ | InsideOutSanta 2 hours ago | parent [-] | | This exactly. Moving from GitHub to Forgejo was much less of a problem than we thought when we just told Claude to figure everything out. Kinda crazy, but it worked. |
| |
| ▲ | Bnjoroge 3 hours ago | parent | prev | next [-] | | Forgejo, and Gitea are decent alternatives if you want to fully move off Github! If you are hoping to move away from only Github Actions, I've been working on a ~drop-in replacement for their control-plane + runner for the past few months. Each each job runs in a cross-platform vm so you can run your ci locally and update your checks. It's still beta, so I certainly wouldnt use it in prod just yet, but it's great if you want to run some of your CI locally. it allows me to extend it where possible to introduce some new features like JIT pipelines, improve performance or security. https://github.com/preloopdev/preloop if anyone's curious! | |
| ▲ | msdz 3 hours ago | parent | prev | next [-] | | The alternative is, I’m guessing, probably any other hosted git + CI/CD service that doesn’t offer a free tier to zillions of users (human and LLM alike) so that their computing capacity isn’t in a constant arms race with growth. | |
| ▲ | sssilver 3 hours ago | parent | prev | next [-] | | What makes it sticky? I just don't see it. The declarative Github Actions manifests? I imagine very few people use the wiki, because it's so subpar. The issues are a problem, but this is why I always thought issues should be a core part of the Git repo, checked into it, sort of how Fossil does it. And so should the wiki, for that matter. | | | |
| ▲ | nwmcsween 2 hours ago | parent | prev | next [-] | | Absolutely anything? GitHub slowly became a standard through market share. Concourse CI is much better, in fact nearly anything is better than GitHub as it doesn't have first class container support. | |
| ▲ | embedding-shape 3 hours ago | parent | prev | next [-] | | Self-hosted $whatever, managed your way. CI/CD doesn't have to be dog-slow or ever go down when you need it, only when you do your upgrades yourself, as god intended it. I personally use Forgejo + their built-in Actions stuff (mostly GitHub compatible), have one old Linux desktop as a server which runs Linux + Windows runners, one Mac Mini for macOS/iOS builds, using NixOS so I don't have to remember how things work, and it just keeps running 24/7, always there when I need it. | | |
| ▲ | tracker1 3 hours ago | parent | next [-] | | There's something to be said for the social availability that github brings. I'm far more likely to submit a pr for a bugfix to a project on gh than a random self-host site. You can do your own ci/cd runner system, but you still rely on webhooks or triggers, when you aren't hosting the code. | | |
| ▲ | embedding-shape 3 hours ago | parent [-] | | > There's something to be said for the social availability that github brings. I'm far more likely to submit a pr for a bugfix to a project on gh than a random self-host site. Yeah, that used to be the single favorable argument for GitHub, nowadays I think it might be the opposite, at least personally. If the contributor can't be arsed to join the community proper (and maybe get a sign-in code to their email, or whatever login you use), I think I prefer life without their drive-by PRs. |
| |
| ▲ | sleepybrett an hour ago | parent | prev [-] | | Here is the thing, we have self-hosted runners running behind our firewall and even those go down for these kinds of outages. All the jobs just sit in the queue and are never sent to our self-hosted setup at all. |
| |
| ▲ | outworlder 3 hours ago | parent | prev | next [-] | | See, my main issue is with Actions. If that breaks, there's little alternative for the more complex flows. Minor infrastructure changes could be done manually through AWS, but complex deployments would have to be replaced. That's a big task even with AI. The pull request system and related machinery is useful, but Gitlab and the ilk can do the same. Even self-hosted solutions like Gitea. But I don't particularly have a problem with that living in Github. If there's an outage, break glass, force push to main. The actual Git repository hosting part is trivial. If that breaks, Git is distributed anyways, push and pull to any random remote storage. All you need is SSH. The main headache is that automations would break as they are usually hard-coded to a given origin. That is solvable in a pinch, and we have setup mirrors in the past. It really depends on your use-case and how much you are tied to them (Github authentication?) | |
| ▲ | EngineeringStuf 3 hours ago | parent | prev [-] | | swarmfile was built to deal with this type of problem |
|
|
| ▲ | lenerdenator an hour ago | parent | prev | next [-] |
| So they're giving you random free coffee breaks of varying length? Quite the perk. |
|
| ▲ | boxed 3 hours ago | parent | prev | next [-] |
| > Github seems to have degraded measurably after they got acquired. As is tradition. Their explosive growth caused by LLMs seems like it must be a much better indicator than the acquisition. I bet Microsoft are losing a shit ton of money on GitHub right now and seriously questioning the decision to buy them. |
| |
| ▲ | maccard 3 hours ago | parent [-] | | No, their decision to focus on a move to azure is the indicator, as the reliability issues started shortly after that, but before their boom in agentic commits |
|
|
| ▲ | shadowgovt 2 hours ago | parent | prev | next [-] |
| It's correlated and possibly semi-causal, but IIUC the real issue GitHub is suffering from is they don't have a clean mechanism for filtering AI-generated activity, which is growing exponentially and putting significant strain on their backend in new ways that didn't show up before. Many AI solutions use PRs, for example, as a plan-of-record journal, which creates very different access patterns from how humans use GitHub. They could, hypothetically, filter bot traffic; I believe they are probably avoiding that solution because they don't want to be, as a market position choice, "anti-AI" (and you could blame the acquisition on this; I'm not sure pre-acquire GH wouldn't have responded to the AI craze with "LOL, no" and instituted a "no agent-driven PRs" policy). |
|
| ▲ | rfgplk 3 hours ago | parent | prev | next [-] |
| They haven't degraded, their service is being hit by an enormous amount of traffic. PRs/commits are what? Doubling every quarter? |
|
| ▲ | FLeXMurphy 3 hours ago | parent | prev | next [-] |
| >Github seems to have degraded measurably after they got acquired. As is tradition. This old chestnut. |
|
| ▲ | sleepybrett an hour ago | parent | prev [-] |
| Same, they seemed to have a moritorium after there last flurry of debacles, but clearly they are letting their engineers deploy slop again. I will never trust github at any other company ever again. |