Good. That's a gain, not a loss. We're better off without social media psyop manipulations and subscription services bolted onto our code repositories. We're better off without a giant evilcorp getting wholesale access to a large majority of the world's source code.

"Protocols, not platforms" should have remained the philosophy of the Internet. Alas, we are here.

> spread across a bunch of disparate places and services.

This is the way. Stop centralizing everything.

Luckily we already have a distributed version control system that can take over github: https://fossil-scm.org/home/doc/trunk/www/index.wiki

I suppose people can use git too if they ever figure out how to send emails from a dedicated client.

At least this will wake up more people to the existing alternative efforts such as Forgejo and its progress with federation protocol.

https://forgejo.org/2023-01-10-answering-forgejo-federation-...

And like Twitter most people will keep using it anyway?

It is, but a single point of failure is itself also an unstable equilibrium, given that a lot of things have to continuously go right for its continued existence in terms of governance of all kinds (technical, moderation policies, commercial/pricing etc.)

I think that especially when it comes to hosting, competition is good. Everything centered on github was the bad state previously. I remember people on HN claiming they wont hire someone without github account.

This splintering should happen more often.

Twitter is unstable? I can't remember the last time it had an outage.

I think that, exactly how 95% of the people that left Twitter eventually returned, we will see a similar outflux of people from Github and the vast majority of them will quietly be back in a few months.

Fuck the community, nobody needs that, especially in a version source control system. Thinking otherwise is just dweebs' eternal longing for belonging, or someone has some bullshit to sell to the unsuspecting dirty masses.

I’m willing to give them a break as I’m assuming they have a lot of scaling problems due to the influx of LLM assisted coding.

But maybe I’m wrong?

I deployed Forgejo on my homelab yesterday. Now GitHub is just a free backup service.

I guess they're quite happy with that TBF, it doesn't seem like they're that interested in being a git forge anyway.

Atleast you're getting the unicorn. I am getting 404 on valid links. For a moment I thought the repo is gone.

https://imgur.com/b0Xj3Z6

They should replace the unicorn image with a horse or at least a zebra during ongoing incidents.

Because any price at all will immediately cause users to shift to another platform, and GitHub's value is that it is _the_ place to put your code on the internet.

The people with the spreadsheets have determined the value of this loss leader to be too enormous to shut it off.

They also sell the machinery to produce the slop (through OpenAI at least).

If they make it harder to push slop on github, people might treat it as an incentive to generate less slop. Less tokens sold. Bad. Very bad.

The github paying customers are not going to move away.

Plus, it's probably genuinely hard to scale. And they can't throw compute at the problem - all compute is already going to the LLMs.

Microsoft executives have an incentive to pretend that AI is great and scaling it has no problems

They would benefit from a HackerNews-esque karma system.

This site's trustworthiness when it goes for pre-aquisition data was disputed here already:

https://news.ycombinator.com/item?id=48086786

I was there before 2018, their reliability was never that good.

I still remember that they launched GitHub Actions in 2018 and back then I was thinking: They can't even make the website reliable, surely this will only cause them more problems.

Fax. Those falling for the load/capacity excuse are gullible.

GitHub's historical uptime page is fake, they backfilled it with 100% uptime going back to 1996: https://www.githubstatus.com/uptime?page=200

The whole point of issues is that users can create them when they don't have repo write access.

Undoubtedly: <https://news.ycombinator.com/item?id=22833037>

Check out Haxxy, it has a lot of fresh ideas related to issues/pr management

https://youtu.be/iBDUTaKWVdU

That canot happen if the market becomes monopolistic, with every bigtech out there buying every startup worth a penny. The erosion of anti trust in the US created this monstrosity.

https://vimeo.com/355556831

We don't care. We don't have to. We're the phone company.

I think AI is going to change this. Once you've built around github you feel locked in, since moving all your CI and other actions out of there, and your workflows out of there, is labor intensive. Now you could migrate from github to another service in a week, maybe less depending on how many workflows you need to move.

A lot of it is inertia I think. I hear people saying that they "just use so-and-so for version control" but really we all have tons of CI/CD build and test pipelines, config, business processes etc in Github (rightly or wrongly).

Competing services can't afford the data storage to compete with Big tech

That's an overly optimistic view of things. For a lot of us GitHub is critical infrastructure, which if it goes down loses us and our customers money.

> Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code

I sympathized with them when they said this a handful of months ago, but then I saw this [0] page that shows how it's been shot for years prior (which tracks with my memory).

I feel for the GH engineers that have to deal with this, especially the SREs. I also don't hate the downtime right now, as I'll make a cup of coffee and do something else. I will say though, I did have a hotfix a week or so ago during the Actions outage, which really was a pain.

You're right that getting angry and ranting isn't the right reaction here, but I do no give them the LLM load excuse. I don't give them an out for having awful uptime during work hours for a product we pay quite a bit for as an org.

[0] https://damrnelson.github.io/github-historical-uptime/

For the amount of money enterprises are paying to GitHub, there is a reasonable (and contractual) expectation of uptime. I don't think my boss would find it to fun if I bailed work to hit the gym just because GitHub was down.

> Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code

Which means they have a pricing problem. Rate limit non-paying users.

Stop blaming LLMs: https://damrnelson.github.io/github-historical-uptime/

Actually, I do have an important fix for a deal we are trying to close that does need to go out. And I am paying GitHub to host this. Once or twice, I can see, but GitHub's SLA is getting worse than just hosting it myself, and that's the entire reason I pay GitHub.

Geez, sorry that the paid service my company paid for is down and I can't do my work.

This has to be rage bait, this is a critical piece of infrastructure for many people. Stuff going down can lead to deployments failing and as you mentioned in many cases emergency hotfix's.

The idea that we should be fine with this unreliability is just amazing. It's not a mental health issue to have problems when important infra fails.

> due to LLM sloppers pushing large amounts of trash code

Which they encouraged by pushing Copilot down everyone's throat

GH shouldn't have promoted LLM usage the way they have if they didn't have the infrastructure to support it. Regardless of how you feel about the code quality (which you have zero evidence of), if GH has made commitments to supporting broad LLM usage, they need to back that up with the proper hardware and without whatever fragile SDLC processes they have.

> To everyone who is angry: calm down. Github’s servers are constantly on fire

Is this rage bait? Isn't Google, Amazon, and all the other services in the world similarly impacted by LLM's? Isn't Claude, OpenAI, etc.? Why can they handle the load but not Github?

What if we're not angry and just perplexed? Do we calm down further or can we still ask questions?

It's every other day now. We pay for a service, we expect service.

what if I do have an emergency hotfix for a production outage though ;-;

This might be naïve but wouldn't the appropriate response be to reduce access/rate limit free/new accounts in order for service to be maintained for everyone else?

> as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code

Github and their parent company are active participants in pushing LLM-driven coding.

Why do you get to tell me how I should feel?

> constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code

I find it hard to believe that github APIs don't have rate limits to handle traffic spikes.

I understand the need to level-head the discussion and calming everyone.

However, your "(you don't)" comment is not going to calm down all the people, who, you know, DO have a hotfix to push now, and DO have an angry customer that could not care less for which part of our infrastructure is breaking _their_ workflow.

The only things would calm everyone down is guarantee that Microsoft would be paying for _our_ SLA breach compensation. But they don't. And I don't think anyone is paying their GH bill with a prorata of the number of time the platform was actually available.

(I'm also aware that the wording of the contract probably clearly says that you should not use GitHub for anything critical, that Microsoft is only a small startup in their garage, that you can't credibly expect 90% uptime anyway, and that it's all the fault of LLM slop ! Bad LLM slop ! Also, please buy our LLMs to generate more slop, please.)

Allowing and accepting the LLM load is a willful choice they made and are making at the expense of their users, including their paying enterprise users.

They could easily tighten things up in that regard, and make a choice that is right for their main users at the expense of The MS corpo mandate/mission. It is a choice to do otherwise.

As a _paying_ customer, I disagree.

Alternatively, don't use GH for critical infrastructure they are no longer reliable, simple as.

Well, they’re a critical piece of infrastructure with terrible stability. I’m in the process of migrating us off GitHub now.

I’m not a Meta fan but it’s interesting that they manage to keep their systems up with an order of magnitude more traffic. GitHub’s uptime is inexcusable.

While I agree in principle, Github with their CI is very critical to many parties.

Why would you take what they say at face value? I have my sources for the fact that this is basically a lie covering up deeper rot.

> due to LLM sloppers pushing large amounts of trash code

hey bud your bias is showing

Or, you know, it's perfectly reasonable and natural to feel angry when a service you paid for gets worse over time.

> Unless you have a emergency hotfix (you don’t)

Oh, we do. Given the sheer number of users, it's almost guaranteed someone is on fire ever time GitHub is down. Statistics is a very charming branch of reality.

No one ever has to commit anything time sensitive.

> increased something like 50x due to LLM

This should not be an excuse since they probably can use AI to fix it /partially sarcasm

it's been trash for years, their status page just doesn't reflect that.

Except I would like to get work done at work and spend time with my family when get home. Taking a walk at work is fine during the outage, but the kids will have to eat dinner without their dad. I'm not okay with that.

Wouldn't be surprised if we lost free private repos because of the slopocalypse. Though most don't seem to be shy about sharing their slop with the world so I'm not sure if that will actually increase stability.

FYI this is an argument to not use Github, not an argument to use it.

In this case, perhaps they shouldn't allow AI slop or promote usage of AI agents...

> go hit the gym or walk outside.

My mind immediately went to this classic xkcd: https://xkcd.com/303/

You know, I am not employed.

I have little time in my day to spend with my daughter. But I also have responsibilities. And one client has decided to bet on github. It's the only client I've ever worked with who has their code on github. EVERY other client has hosted their own gitforge or used bitbucket.

So now I am not angry because some critical piece of US-american infrastructure is down all the time.

I am angry because instead of spending time with my daughter I have to work on this later, because there are due dates and "Well, fucking GitHub was down" ain't gonna cut it.

It has nothing to do with load.

What if you, you know, have a job?

is the infrastructure of the free tier shared with that of paid users? That might be the issue. Otherwise if usage of paid users scales, then it would be fine.

Agreed. This impatience culture is so toxic. Been down less than 30 mins.

i'm surprised the agent didn't take into consideration the http status code

Maybe it's this one: https://downdetector.com/status/windows-azure/

claude also had/has some issues: https://status.claude.com/incidents/zhk4v3yv1lsf

no, stop giving the AI companies free marketing

Sure, but it's not just the reoo. It's PR's, code review and you've got a bunch of actions then it's difficult (not impossible) to deal with those being down.

> We're starting to have the conversation of "do we need to replace GitHub as our CI/CD pipeline?"

Did you have the conversation "do we need to move from $whatever CI/CD to GitHub Actions?" beforehand? What were the arguments in favour?

Worth being upfront with, I'm an employee, but; Buildkite.

Frontier labs and global saas heavyweights run CI through us, plus a crop of open source tooling like Bazel, vLLM, Firecracker, Rails, Ray, Sorbet and Rust.

Shopify, Uber, Airbnb, Canva stresstested the platform ten years ago for scaling issues.

You need to take a little time to design upfront and take advantage of dynamically programmable primitives. It isn't the same ease/convenience as Actions. Bit more available though.

We use self-hosted Github runners, but it's a moot point if all of Github is mostly down as is the case right now. :\

Self-hosted GitLab at work, self-hosted Gitea at home.

If your app is not too large you can do CI/CD perfectly fine from your laptop. Put it in a docker container if you want to ensure it is perfectly reproducible. You still need Github as the origin repo to pull from, but the "git part" of GitHub seems to be (almost) always up and running. (We use "capistrano" for this)

If you're seriously looking at Gitea/Forgejo, I run Fjord, dedicated managed Forgejo, so you don't have to operate it yourself. We can help with migration and runners too. We also run a free public Forgejo instance if you just want to kick the tires first.

ultimately, folks need to stop writing massive YAML files to define their builds. CI should schedule jobs, your build system should define them. my workflow files look like this:

      steps:
          name: Do the thing
           run: nix build .#

Self-hosted GitLab and self-hosted GitLab Runners (Omnibus Docker image)

> What's everyone else using? Self hosted Gitlab? Gitea?

tangled.org!

Self hosted Gitlab at work with local runners

Have you considered Codeberg?

blacksmith.sh just raised tens of millions on the core premise that self-hosting CI/CD is hard [0].

they've had multiple outages in the last week, so i guess they are proving themselves correct.

in practice, giving ssh/sudo to AI (codex) and telling it to setup ephemeral self-hosted runners on your own host VM (qemu), takes about 30 minutes and works extremely well. it bangs out all the shell scripts. moved my own business over to it. zero problems.

you might even find it is kind of neat too. you can customize your VM with software pre-installed. speeds up your builds quite a bit.

[0] https://www.blacksmith.sh/blog/blacksmith-seed-announcement

Self hosted Gitea

TeamCity

Doing the same thing, and I'd join you in this consulting haha. It's a breath of fresh air to know what your CI is actually doing

Sounds interesting but I'm just going to do the same thing as you (self host).

I work at a startup with ~20-30 devs. We're talking about a project like this. I'm not sure weekends would work for an actual company, though maybe it depends. I think there would be an appetite for this. From my POV - the main areas of interest are how you can roll this out to a team gracefully without breaking everyone's workflows for multiple days. Good luck

It means it doesn't count towards their uptime at all, I guess, because all their uptime stats are very high for a service that is down more than anything else I use

Huh, they usually do? I'm not a full-time developer though so I'm surely biased towards doing the obvious quick fixes or already-discussed things, but discussing how to solve a problem before throwing something at the wall seems like the normal thing to do for me

Or is their point about abandoned projects where the dev never clicks release after merging, or even never merges?

GitLab is probably the most reasonable alternative. I migrated and it took me some weeks to adapt to the new CI, but for my use case it's basically equivalent in features.

I started with the cloud version, then moved to self-hosted because it wasn't a big effort for our team. We also reduced our monthly invoice by about 50% after the move.

I’m hosting my own https://tangled.org knot and spindle, and also hosting and building my own Frontend browser for https://forgejo.org looking forward to open sourcing it in the future for others to enjoy.

For people who want a similar experience today, I’d recommend trying Codeberg (An official instance of Forgejo) though. However, keep in mind they voted in July to ban mostly AI generated repositories. You can read about that here: https://blog.codeberg.org/protecting-our-floss-commons-from-...

* GitLab

* Forgejo

Those are my favorites. Both you can self-host for your company, including CI/CD.

I am building an MIT-licensed compatible alternative to GitHub Actions at github.com/automata-ci/automata. Very early in development but it already runs some of our internal workflows.

Copy/paste of a comment I posted a couple months ago:

I am not trying to make any excuses for GitHub, however I recall one tweet that made my eyes bug-out:

> Yup, platform activity is surging. There were 1 billion commits in 2025. Now, it's 275 million per week, on pace for 14 billion this year if growth remains linear (spoiler: it won't.)

> GitHub Actions has grown from 500M minutes/week in 2023 to 1B minutes/week in 2025, and now 2.1B minutes so far this week.

https://x.com/kdaigle/status/2040164759836778878

https://xcancel.com/kdaigle/status/2040164759836778878

---

edit: see a counterpoint I posted down-thread.

Their infrastructure is buckling under increased demand from AI generated code.

Imagine the software development lifecycle is a river blocked by several dams- necessary processes that also slow the water’s volume and speed. Writing the code was a very big dam upstream, and successive dams didn’t need to handle a lot of water. With that dam busted wide open, the torrent of code is inundating downstream processes. GitHub happens to host many of them.

Earlier this year the GitHub COO shared a post on Twitter with explosive numbers they were facing. I cannot seem to find it at the moment. (Edit: shared in other replies to this thread.)

I think the most charitable theory is that they're seeing rapidly growing usage (more users churning out higher volumes of code, via LLMs), and are struggling to scale to meet demand.

Alternative theories include "azure sucks" and "they're vibecoding everything now".

They say it's the skyrocketing load because of AI tooling

Many people suspect it's the year-long migrating to Azure and all their products instead of the things they have been using.

Microsoft bought it, https://damrnelson.github.io/github-historical-uptime/

People will say LLMs but the issues with GitHub long predate LLMs. It all started when Microsoft acquired them.

The CTO discussed it a few months back: https://github.blog/news-insights/company-news/addressing-gi...

Totally not because of their movements to Azure.

Github's, COO Kyle Daigle, claimed that the number of commits has increased 14x since last year. So shear scale has something to do with it. I don't know of too much software already operating a planetary scale that could easily absorb a 14x bump in a year.

AI-Agents are hammering GitHub.

Whatever the actual increase in productivity may be, the commits, clones, pushes, action runs, etc. are skyrocketing.

Lately? GitHub has always been bad with downtime. It hasnt got worse just more visible as people do more CI now than before.

Its all vibes man

Microsoft simply can not make software anymore. I worked at S&P Global and when you work at a global-mega corp, you quickly see why they lose the ability to create anything and just acquire companies to grow.

Interestingly enough if you access the PR number directly it still works.

Not for me. Unicorn error.

gh CLI works for me

Why excluding GitLab?

By switching to Azure....

Pass the buck. There's no one to blame but yourself if you're self-hosting.

"Yeah we're unable to offer services, but so is [X BIG CORP] and [Y BIG CORP]. It's an industry outage, nothing we can do!"

Ideally something like IPNS to have a mutable ref based on (repo, pubkey, branch) to resolve a commit hash, a DHT for who has what commit hashes, and then just p2p transfers. You could still have a github or gitlab for issues and CI, but code (and maybe issues) could all be p2p like git is already designed for. Your forge could also act as a peer for code cloning, of course.

Who's we? Your company?

Otherwise there's no "we", Git is decentralized. Everyone can use whatever they want.

For us the uptime issues have been annoying but not consequential enough to justify the work / risk of moving.

I suspect even poor uptime for many services is a cosmetic problem for most users.

I think the clear reason is even with their unreliability, the cost of migrating off of GitHub for _most_ places is not worth it, and so companies don't / won't (yet)

Its hard to get lemmings to change course.

Biggest problem with Fossil is immutable history. That’s a design choice, and I respect them for that, but it’s not the right choice for me.

As someone who maintained a product that was, until recently, on Azure DevOps, stay away. Like, really really far away. There's nothing positive I can say about it.

The worry I would have is, is anybody at MS still working on it? It is barely updated anymore:

https://azure.microsoft.com/en-us/updates?filters=%5B%22Azur...

That's not necessarily a bad thing for stability, but long-term I would not be surprised if it gets EOL-ed, or radically shrunk to just a basic Git mirroring service.

For your personal projects, do they _need_ to be on a remote git server with a full GUI?

Microsoft has thrived like this for 40 years

I was vaguely remembering that there’s http://githubstatusstatus.com, but it isn’t very helpful unfortunately.