> We've identified an issue with a database primary and are failing over to a replica immediately

This is why it's hard to take GitHub seriously. How can a single database cause an outage for everyone? This is amateur stuff. Have they no sharding or partitioning internally? Paying customers should not be impacted in the same way as free ones are.

▲

ZiiS 8 hours ago | parent | next [-]

2.9B commits per month; 100M action runs per day; I think they probably have some sharding.

▲

ferguess_k 9 hours ago | parent | prev | next [-]

I wonder what is this database, and why it is hard to fall-over automatically.

▲

inigyou 9 hours ago | parent | next [-]

RDBMS replication and failover is way more difficult and manual than anyone would like. You can't just set up two postgres, tell them they're clustered and have it basically work; at a minimum you have to design the client to somehow know which one is currently the master, or use some sort of proxy (which becomes its own SPOF).

RDBMS integrity basically requires that one master server is responsible for the whole data set and other servers may replicate from it. And it usually doesn't wait for a quorum of replicas, just for one, because the design is to recover from a hardware failure, not a network partition, although that could be fixed at the cost of increased latency.

	▲	ferguess_k 8 hours ago \| parent [-]
		Thanks! I didn't get the chance to manage RDBMs but that's good to know.

▲

croemer 8 hours ago | parent | prev [-]

Possibly vitess from the latest update:

> primary failover briefly improved performance but did not fully mitigate, we've throttled inbound traffic and are investigating upstream Vitess issues

	▲	ferguess_k 8 hours ago \| parent [-]
		Thanks!

▲

rkozik1989 9 hours ago | parent | prev | next [-]

Did you not read it? Just because there's a database primary doesn't mean there is 1 primary database. There's likely man redundancies and they have issue with how they're allocating traffic to them which is in turn causing an issue with how much traffic redundancies are receiving.

▲

inigyou 9 hours ago | parent | prev [-]

Why shouldn't it? Most companies run on a single database server. If they can immediately fail over to a replica, that's doing it right.

Maybe you expect that part of GitHub to have a scale where a single database can't handle it, but evidently that isn't true.

We can criticise them for not splitting up free and paid customers but again, most companies don't do that.