Remix.run Logo
WestCoader 4 hours ago

Nothing is pissing me off more than GitHub's stability going down the tubes RIGHT as work is migrating everything, and I mean everything, from CircleCI to GH.

The wildest thing is that Azure Repos/Pipelines was better than this.

Their one caveat is also that they are still migrating it to Azure infra, so it's possible that's still in a one foot in one foot out kinda scenario, from what I've heard. But, this isn't inspiring confidence.

nicoburns 3 hours ago | parent | next [-]

They're claiming a huge increase in traffic due to vibe coded projects. It might just be an excuse, but it certainly seems plausible to me.

motbus3 3 hours ago | parent | next [-]

Could be. But 99% of the repos are static garbage with no PR nor actions.

They mentioned they have some elasticsearch reindexing going to, I would guess they needed to regard or move stuff and something didn't work well. But if I understood it right they mentioned the PRs ES index which they didn't shared proof increased as the number of repos.

It might be anything. It seems they lost huge chunks due to layoffs and structural changes and MS which has the reverse golden Midas touch.

This is just pure speculation but also now there is no reason for MS to keep GH working. They absorbed all code they wanted. Now they can let it burn. Would be even better for them if that happened

jonfw 3 hours ago | parent | next [-]

> Could be. But 99% of the repos are static garbage with no PR nor actions.

But the 1% of repos that do have PRs and actions are likely going to be seeing enormous increases in volumes

I have been a part of two very large companies with self hosted gits and I've seen enough to be confident that this is an incredibly hard thing to manage

fourseventy an hour ago | parent [-]

Ya but they are owned by freaking microsoft and have billions of dollars and employees to throw at the problem. The outage problems shouldn't be happening period.

jonfw an hour ago | parent [-]

Easy to say that! Some problems are legitimately hard to solve though. Github is likely seeing usage patterns that have never been seen before and I bet some of these failure modes are novel

If you are at the limits of your architecture you may need to re-write things, and if you are rewriting things you can not arbitrarily speed that up by throwing dollars at it.

motbus3 11 minutes ago | parent [-]

It is not like MS is involved with AI and say they can make anything in minutes with AI too

giancarlostoro an hour ago | parent | prev | next [-]

At that point, make it lazy indexing? Who cares that I can't find a repo that was made 10 seconds ago, or even 15 minutes ago? No seriously, who cares? Search to that level of nuance is not mission critical, I don't care what anyone says, you'll live if you wait another 15 minutes or even an hour. Their search has been terrible since their last major set of search changes where they overhauled it completely either way.

parthdesai 3 hours ago | parent | prev [-]

Serious question, have you been part of an org that had to scale orders of magnitude very quickly?

Anyone who has been part of that journey knows how painful it really is. A lot of times the systems to fail at all levels, and you have to redesign it from the first principles.

dijit 2 hours ago | parent | next [-]

> Serious question, have you been part of an org that had to scale orders of magnitude very quickly?

I have, but it depends what you mean.

Scenario 1: e-commerce SaaS (think: Amazon but whitelabel, and before CPUs even had AES instructions); Christmas was "fun".

Scenario 2: Video Games. The first day is the worst day when it comes to scale. Everything has to be flawless from day 0 and you get no warning as to what can go wrong.

Yet, somehow, I managed to make highly reliable systems.

In scenario 1; I had an existing system that had to scale up and down with load, this was before there was cloud and hardware had a 3-4 month lead time, so most of the effort was around optimising existing code, increasing job timeouts and "quenching" sources that were expensive. We used to also do so 'magic' when it came to serving requests that had session token or shopping cart cookie.

In scenario 2; we have a clean-room implementation and no legacy, which is a blessing but also a curse, there's no possibility to sample real usage: but you also don't need to worry about making breaking changes that are for the better. With legacy you have to figure out how to migrate to the new behaviour gradually.

So, pro's and con's... but it's not like handling huge load hasn't been done before, computers are faster than they ever have been and while my personal opinion is that operational knowledge is dying (due to general distain for people who actually used to run systems that scale: not just write hopeful "eventually consistent" yaml that they call deterministic) - the systems that do exist today hold your hand much better than they did for me 20 years ago.

And I ran 1% of web traffic with an ops team of 5 back then. So, idk what's going on here.

EDIT: Likely people are flagging me because I sound arrogant (or I hurt their feelings by talking bad about YAML-ops), but all I am doing is answering the question presented based on my experience.

Dwedit 2 hours ago | parent [-]

I think you meant "green fields" and not "clean room"? Clean room refers to reverse engineering an existing program to create specifications, then having another team implement the specifications without legal risk from involving the original.

dijit an hour ago | parent [-]

Yes I did, sorry! You are right. :)

HWR_14 3 hours ago | parent | prev | next [-]

Is GitHub scaling by orders of magnitude though? That would be an insane increase at this stage of their lifecycle.

jodrellblank 2 hours ago | parent | next [-]

They say it is at least one order of magnitude[1]; "our plan to increase GitHub’s capacity by 10X in October 2025 .. By February 2026, it was clear that we needed to design for a future that requires 30X today’s scale."

[1] https://github.blog/news-insights/company-news/an-update-on-...

ori_b 2 hours ago | parent [-]

Note the lack of concrete numbers on how much they have scaled. Somebody may have just asked an LLM for projections.

codechicago277 26 minutes ago | parent | next [-]

https://gitcharts.com shows ~310 million public repos today, vs. 250 million in April 2025 (according to the wayback machine).

Large increase, but nothing existential.

Barbing 2 hours ago | parent | prev [-]

Would Microsoft lawyers OK that?

GitHub would have obligations to MS investors to make accurate projections just like Microsoft itself, right?

HWR_14 39 minutes ago | parent [-]

I don't think it's an issue here. If the investor relations people put it out, it would be. But in this case it is closer to marketing.

nicoburns 2 hours ago | parent | prev | next [-]

I wouldn't be surprised. Have you not noticed the sheer volume of slop being posted everywhere these days? Almost all of that is hosted on Github. And some of those repos have insane commit frequencies.

ambicapter 2 hours ago | parent | prev [-]

If they're suffering the onslaught of ai slop, it's possible.

owebmaster 3 hours ago | parent | prev [-]

> you have to redesign it from the first principles

And that start by layoffing your best engineers, I guess

mitchell_h 2 hours ago | parent | prev | next [-]

They can claim that...but if you've built a public SaaS before you know the job is not to host the software, it's to put rails around people taking it down. They've had since 2008 to build those rails, and they're just now hitting places that take the service down on the regular?

empath75 26 minutes ago | parent [-]

The problem is that they are charging per seat and need to start charging for usage.

AznHisoka 3 hours ago | parent | prev | next [-]

Yep, definitely more traffic and also more new Github repos being created, with a pretty huge spike the last 2 months [1]

[1] https://bloomberry.com/data/github/

reaperducer an hour ago | parent | prev | next [-]

They're claiming a huge increase in traffic due to vibe coded projects. It might just be an excuse, but it certainly seems plausible to me.

I simply do not care.

Customers pay for a service. If they don't get what they paid for, it's perfectly reasonable and normal to go elsewhere.

Why do people on HN keep apologizing on the behalf of trillion-dollar companies?

pixl97 an hour ago | parent [-]

I mean, what will happen here is people will go to other services and they'll get overloaded too.

Self hosted is probably the way to go, but hardware prices are insane currently.

u_fucking_dork an hour ago | parent | prev | next [-]

Probably true. GH Enterprise Cloud is mostly 100% uptime over the past 90 days.

twoodfin 3 hours ago | parent | prev [-]

I’d be shocked if this wasn’t the reason.

dwedge an hour ago | parent | prev | next [-]

Two weeks ago I had a commission to explore migrating from selfhosted gitlab to github for better AI integration. Last night that project was cancelled due to github outages and we're going to upgrade the self hosted server instead. I'd be tempted to use something like forgejo but there are a dozen devs and honestly I've only ever used it solo.

olafmol 39 minutes ago | parent | next [-]

Out of interest: what does "better AI integration" mean? Any specific functional or non-functional requirements?

dwedge 25 minutes ago | parent [-]

I didn't challenge them on this but it's because of Claude integrations with github. I'm not sure what that gives them over just running it against the codebase, but I didn't want to lose the opportunity to finally move them from that EoL server

whstl an hour ago | parent | prev [-]

I would try to sell it internally. The interface is not that different, and I had good experiences myself stability-wise.

ori_b 2 hours ago | parent | prev | next [-]

You can cancel the migration, no need for sunk cost fallacy.

bisby 2 hours ago | parent [-]

"You" can't necessarily do anything (you would be making a lot of assumptions about the influence this person has over the decision making process).

"Someone" can cancel the migration. "Someone" just won't.

ori_b 2 hours ago | parent [-]

https://en.wikipedia.org/wiki/Generic_you

Muromec 3 hours ago | parent | prev | next [-]

Azure repos are kinda fine. It's really basic and there is nothing to break. I actually really really like their ticketing thingy for the same reason. It has the necessary stuff and the management types can't add a million of fields to it and annoy me with reporting, burndown charts or what not.

stackskipton 2 hours ago | parent | next [-]

Yea, I have Azure DevOps with free action minutes and I’ve started using it a ton more since it avoids all GH outages.

Quothling 2 hours ago | parent | prev [-]

It has an annoying bug where approving PR's from the cli won't delete branches when you squash commit, while clicking the button in the UI does it perfectly fine. It's been a bug for a while (as in several years), and if you find something like that, don't expect it to ever be fixed. As a whole it's not a bad tool though.

As you say it's limited, but that can be both good and bad.

jeffwask an hour ago | parent | prev | next [-]

We ended up an Azure Pipelines kinda by default because it was there and mostly paid for with the intention of later migrating, but it's been fine. Boring but stable and functional.

olafmol 41 minutes ago | parent | prev | next [-]

Out of interest: what VCS where you using with CircleCI? (as CircleCI is VCS agnostic)

rufasterisco 2 hours ago | parent | prev | next [-]

i might be connecting unrelated dots, yet when i read "migration to Azure" this came back to me

https://news.ycombinator.com/item?id=47616242 https://isolveproblems.substack.com/p/how-microsoft-vaporize...

biglyburrito 3 hours ago | parent | prev | next [-]

I'm on the other side of the fence. We're just about done migrating from GitHub to GitLab (self-hosted) and it's been refreshing to DGAF about any of the GH outages I read about.

embedding-shape 3 hours ago | parent | next [-]

Similar boat myself too, finished moving all important stuff from GitHub to self-hosted Forgejo with cross-platform builds. Not only do I avoid all the downtime stuff, but E2E builds also takes ~20% of the completion time it used to take, since now my runners have dedicated hardware hosted at home.

rmaus 2 hours ago | parent [-]

To maintain a fair comparison, GitHub has supported self-hosted runners for several years (maybe that doesn’t work for your specific usage, for whatever reason).

embedding-shape 2 hours ago | parent [-]

> To maintain a fair comparison, GitHub has supported self-hosted runners for several years

Yeah, tried that first, as I didn't want to move to Forgejo, I just wanted to keep working when I wanted to work.

The GitHub runner on Linux seemed fine, but the ones for macOS and Windows seemingly did something that made them a hell lot slower than even running VMs and then executing stuff inside those. I'm not sure what the runner is doing, if there is some built-in sandboxing or what not for those platforms, but it wasn't feasible to rely on for me as the builds took way too long time.

anilakar 3 hours ago | parent | prev [-]

We were on self-hosted Gitlab but after a merger were forced to Github. Navigation feels painful in comparison and basic features such as commit graph are now behind more expensive tiers.

plqbfbv 13 minutes ago | parent | next [-]

> We were on self-hosted Gitlab but after a merger were forced to Github. Navigation feels painful in comparison and basic features such as commit graph are now behind more expensive tiers.

Same experience here. Add to that that even on Enterprise tier:

- 1 Enterprise : 1 namespace - although you can segment it with Orgs, we were advised not to do it because we're too small (~2k people) (GL: groups, subgroups, sub-subgroups, ...)

- SSH deploy keys are singletons across the entire instance and repo-bound (and Weblate for instance can only use its own key), so you need a service account for that (GL: instance-wide SSH deploy keys that you can activate in specific repos)

- GHCR only really supports classic PATs for authentication ( https://docs.github.com/en/packages/working-with-a-github-pa... - GL: proper deploy keys properly inherited throughout the hierarchy)

So all in all the experience so far is a huge step-down. I really liked pinning commonly accessed pages in the sidebar.

motbus3 3 hours ago | parent | prev [-]

Interesting! I worked with Gitlab and I also thought it was quite clunky. If it was not for the stability issues GitHub is fine. Any other alternatives to GH or GL?

strenholme 2 hours ago | parent | next [-]

Self-hosting with open source code:

- SourceHut: https://sr.ht/~sircmpwn/sourcehut/

- Forgejo (used by Codeberg, etc.): https://forgejo.org/

miroljub 2 hours ago | parent [-]

SourceHut never really clicked for me. It doesn't give me anything useful that I don't already have in a bare git repo through a ssh.

Forgejo, on the other hand, is a drop-in replacement for GitHub.

olafmol 36 minutes ago | parent | prev | next [-]

Gitea might be an option also.

jeena 3 hours ago | parent | prev | next [-]

We switched from Bit bucket to Gerrit internally and it was a steep learning curve for the des but it's fine.

At a customer we're implementing GitHub Actions and even on our Dev environment there are so many hickups with GitHub.

Mashimo 3 hours ago | parent [-]

Demo of Gerrit here: https://gerrithub.io/q/status:open+-is:wip

Mashimo 3 hours ago | parent | prev [-]

Jira / Bitbucket / Teamcity.

Might be pricy though.

xp84 2 hours ago | parent [-]

Having used Teamcity for CI I cannot think of a more clunky and hard to use system (compared to GHA, which is what we migrated to).

Insanity 2 hours ago | parent | prev | next [-]

If you’re making this change now, I wonder how the technical leadership evaluated GitHub and its competitors.. and then still landed on GH.

What made it better than e.g GitLab?

Xmd5a 3 hours ago | parent | prev | next [-]

Artifacts - C'mon Wit Da Git Down

https://www.youtube.com/watch?v=Js_Y_q-IkYo

sdevonoes 3 hours ago | parent | prev | next [-]

Why do you care about github? It’s Just another corporation doing what they know best: harvesting money. The software ecosystem can live without github just fine

gonzo41 3 hours ago | parent | prev | next [-]

Mee too. We just did a very similar migration at work it's incredibly frustrating, I've got all my CI ported over and now this.

MSFT should just create slophub.com they'd make money im sure.

Mashimo 3 hours ago | parent | next [-]

Honest question, why are companies interested in hosting on github?

As a private person I use it too as a free hoster, but from work I mainly know self hosted instances of jenkins and TeamCity.

xp84 2 hours ago | parent | next [-]

I think you’ve got it backwards. GitHub is by far the market leader for hosted repositories and maybe for CI too. This is like asking “Why are companies interested in using AWS?”

When one firm is so dominant for so long, the question is more like “Why shouldn’t we just use GitHub like 80% of software companies do?”

The issues they’ve had are almost all very recent. Very few companies have reevaluated that decision, because moving a big and well-integrated part of infrastructure is a huge project that delivers no value to the business. Speculating that you’ll have fewer development-slowing outages is not the most convincing when asking for the budget to do this. Plus, self-hosted isn’t necessarily going to have better uptime - mistakes happen.

I think before Actions, it would have been a lot easier to migrate off GH though. You’d just need to change a lot of repo URLs and find a way to set up webhooks from the new place to poke CI. Now with Actions, a lot lives in GH and in a proprietary flavor that doesn’t just ‘lift and shift.’

Mashimo 2 hours ago | parent | next [-]

> I think you’ve got it backwards. GitHub is by far the market leader for hosted repositories

Maybe, but I never heard about any company using github for internal projects in my real life. For me it was always to go to for open source projects.

Then again it's not a topic that often comes up in my developer circles.

jcgrillo an hour ago | parent [-]

More than half the companies I have worked for use Github. The others used Atlassian tools which were at least as bad from a reliability perspective and much less nice to use (IMO).

jcgrillo an hour ago | parent | prev [-]

> The issues they’ve had are almost all very recent.

It has been bad for at least 18mo, maybe longer? I recall multiple work impacting outages at my previous employer extending back into 2024. Maybe even earlier than that?

ellisv 3 hours ago | parent | prev | next [-]

Most developers have experience using GitHub. The UI and concepts are familiar. The friction for adopting features like Actions is relatively low.

Mashimo 3 hours ago | parent | next [-]

> The UI and concepts are familiar.

I guess, but it's not like you can't learn how to create a pullrequest on bitbucket or how to create an issue on jira as well within a work day?

That seems like the smallest thing when switching to a new company.

> The friction for adopting features like Actions is relatively low.

Yeah, I know almost nothing about the CI integration and actions when it comes to Github. Will look into it. Thank you.

not_ai 3 hours ago | parent | prev [-]

At one point it was also used as signaling that a company was “modern.”

embedding-shape 3 hours ago | parent | prev | next [-]

> Honest question, why are companies interested in hosting on github?

Mostly boils down to marketing and easier to establish a community. Almost every developer has an account there, leading to network effects being much larger, so if you're a new FOSS project, finding contributors and getting your project in front of other's eyes is much easier when you're on GitHub compared to your own Forgejo instance.

With that said, I'd question if chasing "most external one-time contributors" or GitHub stars is the right way to actually run a FOSS project, personally I'd avoid thinking about those vanity-numbers as much as possible and focus on the project, code and contributors themselves.

But, I've literally heard those two arguments for "why GitHub" countless of times over the years.

Mashimo 3 hours ago | parent [-]

Oh FOSS projects I totally understand. It's where I go to too.

But closed source companies surly don't need to establish a community?

embedding-shape 3 hours ago | parent | next [-]

Go with the flow, don't rock the boat and use what developers already know, are probably the most cited reasons I've heard.

I've tried so many times in the past to argue for self-hosted setup that you fully control if you can afford it, things just get so much smoother and if you're a software development company, you probably want to own the software development workflow E2E so you can actually ship as fast as you want.

Esophagus4 2 hours ago | parent [-]

I’ve argued the opposite most of the time in build vs buy. Buy in almost every case unless it’s a real competitive advantage to you.

I know developers love to build, but do you think:

1) self-hosting git provides any competitive edge to the business over letting someone manage it?

2) it provides so much value that you’re willing to fund engineers to build, secure, support this on an ongoing basis?

I’ve found the answer to those is No in both cases.

The same reason you wouldn’t build your own internal chat tool, you’d use Slack. And you wouldn’t bother self-hosting your own Jira or documentation.

Code hosting is code hosting, there’s no difference where it's hosted. There’s no slowdown in delivery with using GitHub - their March uptime was 99.5% which annoys some commenters but it’s fine. That’s 45 minutes downtime per month which is tolerable.

You would spend way more effort and money building a jenky self-hosted solution to end up with a worse result.

oasisaimlessly 13 minutes ago | parent [-]

You don't have to 'build' anything. Just spin up a GitLab docker container. Bonus: If you put it behind a VPN, you never have to worry about updating it.

idkyall 2 hours ago | parent | prev [-]

Usually, at large enough corporations, it's one of two things. Some random project gets open sourced, and it ends up on Github(see, for example, Salesforce) - or, more commonly, some subsidiary or acquisition had github and has either refused to migrate to the internal source system or the hassle of migration isn't worth it.

ryandrake 2 hours ago | parent | prev | next [-]

I don’t know why you would even really need hosted git or why you’d be affected by its downtime. Git is decentralized by design. One node going down should not stop development. You don’t need a “central hub” to keep working.

I guess it’s all the other non-git stuff like issue tracking and other (unfortunately) centralized products on GitHub that causes disruption when they go down.

Weird how GitHub built itself around a distributed VC system and then made all its other services centralized.

Mashimo 2 hours ago | parent [-]

> I guess it’s all the other non-git stuff

Yes, you want to run automated builds, unit test, end to end test, UI tests, make it easy for testers to deploy specific versions / tags to internal server. Also kick off builds for iOS on mac computers. We use Teamcity for that.

Tracking of issues, feature and epics. Maybe also knowledge base / wiki. We use Jira.

And pull requests. Bitbucket.

stephenlf 3 hours ago | parent | prev [-]

Onboarding construct workers is super easy.

2 hours ago | parent | prev [-]
[deleted]
throwaway613746 2 hours ago | parent | prev | next [-]

[dead]

giwook 3 hours ago | parent | prev [-]

Github uptime down to 86% according to https://mrshu.github.io/github-statuses/ (not my website)