Time to rant... This is absolutely unreal.

Even self-hosted runners are impacted.... How can that be?

The cost of this globally has got to be in the hundreds of millions to companies that use CI/CD through GitHub Actions. What if prod is broken and GitHub actions is stalling the deployment of your hotfix? What if this makes your organization miss and SLA and diminish user trust? What if this makes you miss a release that you were contractually obligated to meet? This is happening during peak dev hours on a Thursday (not that it would be acceptable at any other time).

I don't understand how a service this critical to the global technical infrastructure can fail like this at all, let alone for more than a few hours. Like where's the backup generator for crises like these? You can't even use self-hosted runners? WTF? Like how can you not bring your own backup in a crisis event like this?

Not that Microsoft has a good reputation, but holy moly, you'd think they would prepare from something inevitable like this.

▲

carlosneves an hour ago | parent | next [-]

That's exactly the situation I'm in... :crying-laughing:

The fix is merged, but won't deploy... it's been hours

Thankfully it's a batch job, and isn't interrupting production ATM

	▲	chiply 33 minutes ago \| parent [-]
		I feel for you.... This is not a position you should be put in. There's always the escape hatch of running you GHA workflows locally, but unfortunately, despite the existence of packages like `act`, there is no way to fully recreate the GHA runtime locally. Tons of the special YAML syntax just can't (more accurately, "just doesn't") get interpreted by those local actions runners. We never went this route, but at my old org, I always advocated for considering GHA to be wrapper around a single bash script (or whatever script you want to run), as a means of completely breaking out of the GHA hellscape that is programming in YAML, who's turing-completeness is pretty dubious. Unless you have things set up this way, you (the client of GitHub) would have to completely redesign your CI on the fly, run it locally, and then figure out how to get the D compliment of the I to work in a way that is auditable. Fat chance for most teams I bet. Thank god you're dealing with a batch scenario. Silver lining for sure. Still, embrace the anger. What makes my blood boil is that there's millions of DEVs literally crying at the moment worrying about how GitHub's failure to be responsible will put their jobs in jeopardy. And fingers crossed for you my friend. We're at 5+ hours at the time of this writing.... You're batch job may still have a chance!!!

▲

nlightcho 2 hours ago | parent | prev [-]

Counterparty risk is still a thing. This is the cost of convenience, reminds me of the milkman joke in that South Park episode. Maybe a global SPOF owned by people who do not care is not the way to go.

	▲	chiply 24 minutes ago \| parent [-]
		Dude that milkman joke made me cackle out loud. It's hard to draw a direct analogy there, but I feel like it echoes the same sentiment. A soapbox I have is that GHA workflows are scripts that could run on your machine without any of the YAML stuff. Who gives a flying about the DAG or the logs? Which, by the way, if you're willing to walk to the milk store to buy your milk, could be recreated in a much more testable and maintainable way without any of the YAML bs that GHA prescribes.... But DAGs are pretty, and logstreams showing up in a browser application instill trust (for reasons that fly far above the head of yours truly). So people go for that. Pretty DAG, nice logstream; therefore, deliver my milk. All of a sudden.... The CI/CD platform is having its merry way with your SLAs, contract abidements, and hotfix deployments. What a time to be alive. Not to rail on the South Park thing, but the blast radius of this issue also reminds me of the episode where the internet dried up. If this bs with GitHub continues, Parker/Stone will have to make a GitHub episode. How seen would we all feel if that happened?