Insane, we have to come up with contingency plans now for long-duration GitHub outages because we can't safely do deployments. For a service we're paying thousands of $ per year for even though we host runners ourselves...

▲

Salgat 3 hours ago | parent | next [-]

It's funny, when we were acquired they started moving us to Github actions but it seems that maybe we should stay on our old crusty self-hosted Jenkins setup...

▲

cryo32 3 hours ago | parent | prev | next [-]

You should never entirely depend on a third party service for deployments.

Been burned too many times on that one.

▲

999900000999 3 hours ago | parent | next [-]

Ok.

Move to EC2.

Darn AWS is down.

Alright, run it on a Mac Mini in your basement. Ahh dawn, your ISP is having issues. Good thing you have a backup 5G hotspot.

Ohh no, the power is out.

Eventually you have to trust someone else.

GitHub is a tragedy of the Commons. Too many people are using it, and Microsoft isn't willing to handle it correctly.

Feels like a very good business opportunity. Minimum 50k yearly contracts, GitHub with actual uptime. GitPro ?

▲

cryo32 2 hours ago | parent | next [-]

We’re actually moving back to redundant data centres due to all of those problems.

Aggregate risk is too high.

▲

sleight42 2 hours ago | parent | prev | next [-]

It's almost as though GitHub should never have let itself be sold to Microsoft...

	▲	999900000999 2 hours ago \| parent [-]
		I'm sure the VCs who invested in GitHub disagree. This is supposed to be Hacker News! Who is coming up with a startup to fill the gap !

▲

bee_rider 2 hours ago | parent | prev [-]

Maybe we need a split between source management and distribution? The former looks like git[hub] to me, the latter maybe more like a Linux distro repo?

▲

bouk 2 hours ago | parent | prev [-]

We could still deploy manually but it's suboptimal! And we're 'flying blind' without CI runs

	▲	matt_kantor 2 hours ago \| parent [-]
		> And we're 'flying blind' without CI runs You should never entirely depend on a third party service to run your tests, either.

▲

decodebytes 5 hours ago | parent | prev | next [-]

Same thoughts - we use an action to ship to production, its builds an image, pushes it to ECS which triggers a deployment.

We can't be blocked here. Seems silly what we settled on this, but for a long time GitHub had been reliable enough for many years, but things are sliding down the pan as of late.

	▲	mystifyingpoi 5 hours ago \| parent [-]
		Sounds like a very easy process to rewrite in bash/python and have it on hand if needed.

▲

the8472 5 hours ago | parent | prev | next [-]

./deploy.sh

▲

dnnddidiej 5 hours ago | parent | prev | next [-]

It is a control pain

▲

Cthulhu_ 3 hours ago | parent | prev | next [-]

It's always best to be portable - always be able to do builds and releases locally (at least, once you get the keys - it shouldn't be possible by default), then add things like github actions on top as convenience.

▲

yoyohello13 3 hours ago | parent | prev | next [-]

Self host gitlab. If you already host runners it’s not a big lift.

▲

sebmellen 5 hours ago | parent | prev | next [-]

Same here. You’d think they could at least separate out the GitHub-hosted and self-hosted runners, so you’re still able to dispatch jobs if the self-hosted runners are down.

▲

ketzu 5 hours ago | parent [-]

If the job queue is down, that wouldn't help, would it?

On my repo the jobs do not get scheduled on the PRs at all, so I assume that separation wouldn't help for todays issue.

▲

voxic11 4 hours ago | parent [-]

They have the github enterprise domain separated out and its working fine right now https://us.githubstatus.com/posts/dashboard

	▲	anon7000 3 hours ago \| parent [-]
		I’m not convinced they actually do, because GHE on the cloud tends to have the same problems as the main outages. Probably costs extra to be “single tenant” or whatever

▲

sofixa 5 hours ago | parent | prev | next [-]

Depending on how many thousands of $ per year, it would probably be cheaper and more reliable to self-host GitLab. It's better in terms of organisational structure (you can have one, including access and secret inheritance), and (personal view) Gitlab-CI is better than GitHub Actions because it doesn't push you towards a JavaScript/NPM style dependency hell. And it's actually fairly easy to self-hosted, with options from a single machine with an omnibus package that handles everything to a full blown autoscaling Kubernetes deployment.

▲

hsbauauvhabzb 5 hours ago | parent [-]

Sounds good until you see their cvedetails page

▲

lazystone 4 hours ago | parent | next [-]

Hide it behind VPN, so it's not accessible from outside.

▲

PunchyHamster 3 hours ago | parent | prev | next [-]

When you own it you can just limit it into vpn-ed company users, that significantly cuts down on the area that can be hit

▲

sofixa 5 hours ago | parent | prev [-]

I mean, the GitHub Actions supply chain risks and attacks definitely compensate for any GitLab security vulnerabilities you can think of.

	▲	user43928 5 hours ago \| parent [-]
		[dead]

▲

re-thc 5 hours ago | parent | prev | next [-]

> For a service we're paying thousands of $ per year for even though we host runners ourselves...

Wait until you charge you for self-hosting runners.

Oh wait. They already tried.

▲

pluc 4 hours ago | parent | prev [-]

Sure. Don't use GitHub.

You can now hire me as an overpriced consultant instead of paying Microsoft.