| ▲ | ekidd 6 hours ago | |
> Managed databases are a scam. I, too, once believed this. Then I had the displeasure of watching a $10,000 server fail during Christmas travel (about 20 years ago now). A single RAID drive failed. Then, during the rebuild, a second drive failed. Then the RAID controller itself failed catastrophically, losing all the RAID volume metadata. When we restored from backup, we discovered that the sysadmin who had just quit a few weeks before had lied to us about the backup system, and we had no backups. This is the sort of black swan event that happens every 5-10 years. It's an unusually bad event, even by black swan standards, but stuff like this happens. The fundamental problem of self-hosted databases is that you test the happy path every day, but you only test true disaster recovery every 5-10 years. And in practice, this means that disaster recovery will usually fail. With a managed database service, most of what you're paying goes to making sure that disaster recovery works. And in my experience, it does. I've seen RDS database servers fail catastrophically, and completely rebuild in under 15 minutes with virtually no data loss, with almost no human intervention at all. If you care about your customers' data, I think that a reputable managed database is the right move until roughly the point that you can pay for a full time database administrator. At that point, sure, roll your own. But do regular disaster recovery tests, lest you discover that a recently departed DBA has been lying to you. | ||
| ▲ | znnajdla 4 hours ago | parent | next [-] | |
Yeah but even with managed database services you don't know if your provider has invested into proper testing of their recovery so you have to test it anyway. Major services like DigitalOcean have been known to shit the bed with your backups. If you don't test your backup recovery, you don't know if you're screwed even if you're paying for "managed" services. I test my backup recovery several times a month by actually baking into our CI/CD workflow under certain conditions. The entire production database gets restored from backup every week. | ||
| ▲ | throwaway063_1 3 hours ago | parent | prev [-] | |
What about a hybrid approach? You could use a managed db service as a live replica dedicated as a backup only. The queries would go to your local database on beefy hardware, while the replica would just have to be powerful enough to keep up with the WAL stream. | ||