Doesn't post-receive block the push operation and get cancelled when you cancel the push?

At times like this is when I'm so happy I don't work with deploying to a production environment, but rather we release software that (after extensive qualification), customers can install in their environment on their airgapped networks. Using a USB stick to cross the air gap. If we miss a release by a day or thrre, there is enough slack in the process before it goes to the customer that no one will be any the wiser.

Crazy in 2026, but installable software has some pros still, for both the developer and for the customer. And I would personally love if I could do things that way for more things.

This is a must when your systems deal with critical workloads. At Fastly, we process a good chunk of the internet's traffic and can't afford to be "down" while waiting for the CI system to recover in the event of a production outage.

We built a CI platform using dagger.io on top of GH Actions, and the "break glass" pattern was not an afterthought; it was a requirement (and one of the main reasons we chose dagger as the underlying foundation of the platform in the first place)

It’s a hard sell. I always get blank looks when I suggest it, and often have to work off book to get us there.

I generally recommend that the break glass solution always be pair programmed.

100%. We used to design the pipeline a way that is easily reproducible locally, e.g. doesn’t rely on plugins of the CI runtime. Think build.sh shell script, normally invoked by CI runner but just as easy to run locally.

A while back I think I heard you on a podcast describing these pain points. Experienced them myself; sounded like a compelling solution. I remember Dagger docs being all about AI a year or two ago, and frankly it put me off, but that seems to have gone again. Is your focus back to CI?

The problem is that any kind of automatic code change process like CI, PRs, code review, deployments, etc etc are based on having a central git server. Even security may be based on SSO roles synced to GH allowing access to certain repos.

A self-hosted git server is trivial. Making sure everything built on top of that is able to fallback to that is not. Especially when GH has so many integrations out of the box

> a lot of infrastructure quietly assumes GitHub is always available

Which is really baffling when talking about a service that has at least weekly hicups even when it's not a complete outage.

There's almost 20 outages listed on HN over the past two months: https://news.ycombinator.com/from?site=githubstatus.com so much for “always available”.

These days it feels like people have simply forgotten that you could also just have a bare repository on a VPS and use it over ssh.

I mean, this isn't a 'URL returned error: 500' situation for anything that Codeberg provides considering this is an issue with Copilot and Actions.

I used to use codeberg 2 years ago. I may have been ahead of my time.

I rarely successfully get Codeberg URLs to load. Which is sad because I actually would very much like to recommend it but I find it unreliable as a source.

That being said, GitHub is Microsoft now, known for that Microsoft 360 uptime.

I mean... you understand the scale difference right?

Sounds like my Dad, who used to have an uncanny ability to get stuck in elevators. Even got stuck in one with his claustrophobia therapist.

Plot twist: cpfohl works at Github and actually messes with the infra.

do you know the Pauli-Effect? https://en.wikipedia.org/wiki/Pauli_effect

You should be promoted to SRE - Schrodinger Reliability Engineer

Simple solution: do infra work every few months instead of every few weeks.

Just let us know in advance when you want to do infra work from now on, alright?

Surely this would earn you loads of internet street cred.

The #1 priority at GitHub for this year is migrating from their own data center to Azure, any other work that gets in the way of this is being deprioritized: https://thenewstack.io/github-will-prioritize-migrating-to-a...

This is an absurd state they are at! Weekly outages in 2025 and 2026. From developer beloved and very solid to Microslop went faster than I expected

I think GitHub shipping Copilot while suffering availability issues is a rational choice because they get more measurable business upside from a flashy AI product than from another uptime graph. In my experience the only things that force engineering orgs to prioritize uptime are public SLOs with enforced error budgets that can halt rollouts, plus solid observability like Prometheus and OpenTelemetry tracing, canary rollouts behind feature flags, multi-region active-active deployments, and regular chaos experiments to surface regressions. If you want them to change, push for public SLOs or pay for an enterprise SLA, otherwise accept that meaningful uptime improvements cost money and will slow down the flashy stuff.

Unless a major out(r)age forces a change of leadership, expect more slop down our throats.

They get posted to the incident log (https://www.githubstatus.com/incidents/n07yy1bk6kc4) for example https://www.githubstatus.com/incidents/lcw3tg2f6zsd.

There are also monthly availability reports: https://github.blog/tag/github-availability-report/

Codeberg?

and we all know why

Right now the page says Copilot and Actions are affected but I can't even push anything to a repo from the CLI.

It's a multitude of factors but basically they can act like that because they are dominant on the market.

The classic "nobody ever gets fired for buying IBM".

If you pick something else, and there's issue, people will complain about your choice being wrong, should have gone with the biggest player.

Even if you provide metrics showing your solution's downtime being 1% of the big player.

Something like Cloudflare is so big and ubiquitous, that, when there's a downtime, even your grandma is aware of it because they talk about it in the news. So nobody will put the blame on the person choosing Cloudflare.

Even if people decides to go back (I had a few customers asking us to migrate to other solutions or to build some kind of failover after the last Cloudflare incidents), it costs so much to find the solutions that can replace it with the same service level and to do the migration, that, in the end, they prefer to eat the cost of the downtimes.

Meanwhile, if you're a regular player in a very competitive market, yes, every downtime will result in lost income, customers leaving... which can hurt quite a lot when you don't have hundreds of thousands of customers.

Businesses are incommensurate.

GitHub is a distributed version control storage hub with additional add-on features. If peeps can’t work around a git server/hub being down and don’t know to have independent reproducible builds or integrations and aren’t using project software wildly better that GitHubs’, there are issues. And for how much money? A few hundred per dev per year? Forget total revenue, the billions, the entire thing is a pile of ‘suck it up, buttercup’ with ToS to match.

In contrast, I’ve been working for a private company selling patient-touching healthcare solutions and we all would have committed seppuku with outages like this. Yeah, zero downtime or as close to it as possible even if it means fixing MS bugs before they do. Fines, deaths, and public embarrassment were potential results of downtime.

All investments become smart or dumb depending on context. If management agrees that downtime would be lethal my prejudice would be to believe them since they know the contracts and sales perspective. If ‘they crashed that one time’ stops all sales, the 0% revenue makes being 30% faster than those astronauts irrelevant.

To be fair - it SUPER does. Being down frequently makes your competition look better.

Of course, once you have the momentum it doesn't matter nearly as much, at least for a while. If it happens too much though, people will start looking for alternatives.

The key to remember is Momentum is hard to redirect, but with enough force (reasons), it will.

The reality is that consumers don't really care about downtime unless it's truly frequent.

If you count every service together, it's deep into one nine.

https://mrshu.github.io/github-statuses/

Most individual services have two nines... but not all of them.

Github proudly boasts an industry-leading seven 9s of uptime. 49.999999%

"78 incidents in last 90 days" per https://mrshu.github.io/github-statuses/

that's....gobsmacking...I knew it was memeably bad but I had no idea it was going so badly

90 day non-degraded uptime of Github Actions is 98.8% if the official numbers can be believed

Due to a off-by-one error, they are now targeting "five eights". Why else would they migrate to Azure?

They're going to have to start advertising nine fives of reliability.

they were down to a low 1 nine recently

Github service has a better work life balance than many engineers here...

Octocat (The OG github mascot) has a family that it goes to the park with anytime he wants.

Luckily his boss Microslop, is busy with destroying windows of his house and banning people from its discord server.

It's Microsoft. You're supposed to Ctrl+Alt+Del.

Top-down mandates to use AI as much as possible, and to rip up their infrastructure and move everything to Azure.

https://www.windowscentral.com/microsoft/using-ai-is-no-long...

https://thenewstack.io/github-will-prioritize-migrating-to-a...

Does anything running on Azure have an acceptable uptime?

At this point I am thinking of creating a 0 days until github outage website similar to how we had the running joke of 0 days until JS framework dropped.