▲ | nucleardog 3 days ago | |
> Granted, it gets harder and more expensive with increasing scale, but it's a necessary expense if you care at all about business continuity issues. On a personal level, it's much cheaper though, especially these days. I don't go as far as "live mirror", but I've been advocating _for years_ on here and in meatspace that this is the most important thing you can be doing. You can rebuild your infrastructure. You cannot rebuild your user's data. An extended outage is bad but in many cases not existential. In many cases customers will stick around. (I work with one client that was down over a month without a single cancellation because their line-of-business application was that valuable to their customers.) Once you've lost your users' data, they have little incentive to stick around. There's no longer any stickiness as far as "I would have to migrate my data out" and... you've completely lost their trust as far as leaving any new data in your hands. You've completely destroyed all the effort they've invested in your product, and they're going to be hesitant to invest it again. (And that's assuming you're not dealing with something like people's money where losing track of who owns what may result in some existence-threatening lawsuits all on its own.) The barrier to keeping a copy of your data "off site" is often fairly small. What would it take you right now to set up a scheduled job to dump a database and sync it into B2 or something? Even if that's too logistically difficult (convincing auditors about the encryption used or anything else), what would it take to set up a separate AWS account under a different legal entity with a different payment method that just synced your snapshots and backups to it? Unless you're working on software where people will die when it's offline, you should prioritize durability over availability. Backups of backups is more important than your N-teir Web-3 enterprise scalable architecture that allows deployment over 18*π AZs with zero-downtime failover. See, as a case study, Unisuper's incident on GCP: https://www.unisuper.com.au/about-us/media-centre/2024/a-joi... |