Remix.run Logo
jdc0589 4 days ago

this is pretty much the situation I'm in re the storage. I'm perfectly fine running a home server, I already do, but workloads with heavy storage requirements scare me away from it. I don't want to have to think about that at home, and the cost of pretty much anything other than object storage in the cloud is prohibitive, and as you mentioned obj store support is non-existent or hacky and slow with most of these products.

namibj 4 days ago | parent | next [-]

`rclone mount` an `rclone crypt` over a Cloudflare R3 backend of `rclone`? Or if it's sufficiently often "off/idle", take 3 USB HDDs (1~2 years ago I bought an iirc WD MyPassport 5TB for very similar workload) into a RAID-5 and have appropriate off-site backup that you actively check to have successfully gotten the latest daily backup's file contents (check a couple (3~5) random files as well as a few (3~5 ish) critical/database/metadata files) at least every 1~3 months.

Also, as opportunities arise like for example from a major upgrade to local storage capacity, try to fully test a backup restore emulated to "your home burned to the ground while you were at the office/on holiday" conditions every 1~3 years if you can afford to spend the bandwidth for it.

Consider burning in drives for a group-buy you do with local friends if necessary to at least get such a full restore trial every about 3 years. Try as best to consider a trial every about 5 years to be a "cost of doing business" that's not just nice to have but essential to the value proposition of the data archive storing home server.

Oh and yeah, I fully mean to let the drives go to sleep when you're not accessing them through "manual"/interactive means (exceptions are limited-time background queued work with a set override timer, and the daily backup runs, which will also unlock the drives from their regular sleep-doesn't-get-interrupted-for-no-good-reason enforcement; ofc this is all something you do only if you can and feel like you want to: just hunting down rouge accesses/wake-ups happening at odd times by setting up some minor logging of which programs/files/accesses (or at the very least _when exactly_) are causing the drives to wake up is something you could very well get away with). Also take care to ensure they get good airflow: stack them with gaps between and ideally just take a decent but low-cost 120mm fan that you just hook to 5V from USB (if you don't have a fan header laying around) and rig with some cardboard and tape to channel air across your drives. The drives want to be around 30~45 C, consider hooking the smart temperature readout to a kill switch in case of fan failure.

j45 4 days ago | parent | prev | next [-]

Self-hosting seems easiest to think about as a home appliance.

Out of compute, storage, database, networking, etc, which is most preferable to be just an appliance?

It's pretty reasonable to get reliable storage self-hosted without the headache. If a big setup isn't needed, it's reasonably attractive to set up your own storage with reasonable power draw, which can be kept up with more reasonable UPS'.

Just because one can build and run a storage array on their own, doesn't mean it would be the best allcoation of their ongoing attention to maintain and be on call for a daycare for hard drives.

If seamless storage as good (and sometimes better) than a cloud is the minimum, it has to be something trustable, and run like a reliable home appliance needing minimum maintenance.

Lots of folk choose NAS enclosures that have raid mirroring and hot-swap drives built in quite inexpensively using things like Synology or QNAPs. The web admin interfaces on them are reasonable, and it's trivial to poke along with a youtube video to setup a RAID 5-10, and send email notifications how you like if it wanted to bring something to your attention.

Other things that become way more valuable over the years:

- NAS can be configured to backup offsite to the cloud backup of your choice, or another NAS. I know folks running them for 5-10 years and never think about it. Decent NAS with appropriate drives, secured of course. Some people even mail the enclosure to a datacenter and have them plug it in and keep it online.

- If you get a reasonably basic NAS with an intel Celeron CPU, power usage can remain low, but ram can be upgraded on it to run a few services as needed on it, both directly, and as docker images. It's pretty wild.

- If you do consider it, my recommendation is to pick one that has 2 extra drive slots than you need, and start from there. People who buy two bays can outgrow them quick, plus it's only a mirrored raid between two drives. Raid 5 and higher is great, if one drive is starting to have issues, you can just swap it while it's all running and the storage will heal.

Hope that helps. Having data close to crunch can be valuable.

yesnomaybe 4 days ago | parent | prev [-]

it's really not that hard. I've set up backblaze which is reasonably cheap. with the help of AI I was able to setup a permanent cron job that backs up everything from local into B2 using rclone, which client side encryption. It's epic. I haven't looked at it for a while but I do DR test every once in a while a small subset and it works really well. I use postgres as DB and this is the big one to back up daily. Rest is just the increment. Can be further optimised I guess but I'm happy with it.