> Whenever corruption occurred, we had to stop the control plane process on the shard while we repaired or restored the database. This was painful for tailnets on that shard, because their entire control plane disappeared during that recovery window.

Gotta love single points of failure...

▲

arjie 2 minutes ago | parent | next [-]

You don’t need the control plane most of the time. I had a zero downtime headscale upgrade because once the nodes negotiate through the control plane they can talk to each other all the time. The data plane is peer to peer.

It’s problematic because you can’t run connections but it doesn’t stop the world.

▲

kccqzy an hour ago | parent | prev | next [-]

What are some solutions to avoid database corruption being single points of failure? I can’t think of any off the top of my head. I don’t think people typically consider database corruption to be a kind of failure common enough to design for, unless you have unusual requirements.

	▲	AlotOfReading 34 minutes ago \| parent [-]
		The general answer to this is Byzantine consensus, which cryptocurrency blockchains are designed to solve. If your nodes are willing to fail a little more politely (e.g. no lying, immediately crashing, etc) you can use something cheaper like raft/paxos. But yeah, it's a lot cheaper to build a reliable system than it is to be resilient.

▲

spockz 23 minutes ago | parent | prev [-]

The shard was already a way to make it not a single point of failure.