Remix.run Logo
gerdesj 7 hours ago

One of our customers decided to go for off site backups with us - they had been a customer for 15 odd years for other services. Backups are not our core business, its one of the services and I got around to setting them up sooner rather than later.

Two weeks later they had a fire in one of their huge printing machines which caused quite a lot of direct damage due to heat and rendered a lot of equipment broken due to smoke/fumes. It was also, as it turns out, a week before the first COVID 19 lockdown in the UK.

Their backups landed on a XFS file system with reflinks enabled. I cloned their repo and fired up NFS with their VMs running on our gear. It took me another hour to spin up another OpenVPN server (CA etc) for them to use and sort out a few other details (coffee doesn't brew itself).

They ran via VPN out of our data centre for most of the pandemic.

That was an utter triumph but I also have some rather less triumphant stories about backups and lack thereof. Let's skip over those 8)