Remix.run Logo
▲ munch-o-man 4 hours ago

good thing all my data with this is encrypted from my end haha. Also, just gonna pimp out https://github.com/afreidah/s3-orchestrator shamelessly here...if you put it between your clients and the backend it is easy to have your data in multiple places and a cloud migration is as simple as having your clients make requests to b2 through s3-orchestrator, add your new backend you want to migrate to in the config and set the replication-factor to 2, your clients keep writing with no idea what is going on, meanwhile a replication worker is copying everything from b2 to the new backend without taking downtime. Once replication reaches zero just drain the b2 backend so write stop going to it, and let it safely unwind itself, then remove it from the config. migration done, basically zero downtime other than a quick service restart.

▲leomos 4 hours ago | parent [-]

That's amazing, thanks for making it!

▲munch-o-man 36 minutes ago | parent [-]

what is funny is it started as a project so I could do free offsite backups of some homelab stuff and I didn't want to spend money and started thinking that if I could sign up for every always-free tier s3/blob provider and "combine" their space it might actually be an amount that is usable...but it needs to be free...so I had to build quota tracking/enforcement options for storage_bytes and monthly api/ingress/egress so I could make sure I never accidentally spent money rofl.

at that point I had spent a lot of time on it and it sort of outgrew my homelab into it's own thing.

btw, contributors are very much welcome, be it issues, missing features/functionality, complaints, pull requests, your favorite chili recipe...I am sure there is stuff I'm missing or has bugs that aren't uncovered by my personal usage of it.

EDIT: While it is no longer just a hack to get free offsite backups and is now a production grade s3/blob data orchestration layer, I still try and keep the original spirit in all decision making and do my best to make sure every operation pays the tech fascists the least amount of money possible haha.