Remix.run Logo
newqer 6 hours ago

Just make sure you have you have a bot network storing the information in with multiple accounts. Also with with enough parity bits (E.g. PAR2) to recover broken vids or removed accounts.

compsciphd 5 hours ago | parent | next [-]

par2 is very limited.

It only support 32k parts in total (or in reality that means in practice 16k parts of source and 16k parts of parity).

Lets take 100GB of data (relatively large, but within realm of reason of what someone might want to protect), that means each part will be ~6MB in size. But you're thinking you also created 100GB of parity data (6MB*16384 parity parts) so you're well protected. You're wrong.

Now lets say one has 20000 random bit error over that 100GB. Not a lot of errors, but guess what, par will not be able to protect you (assuming those 20000 errors are spread over > 16384 blocks it precalculated in the source). so at the simplest level , 20KB of errors can be unrecoverable.

par2 was created for usenet when a) the size of binaries being posted wasn't so large b) the size of article parts being posted wasn't so large c) the error model they were trying to protect was whole articles not coming through or equivalently having errors. In the olden days of usenet binary posting you would see many "part repost requests", that basically disappeared with par (then quickly par2) introduction. It fails badly with many other error models.

catlikesshrimp 9 minutes ago | parent | next [-]

you can split files so you can have more par blocks (100GB in 100 1GB parts 32k blocks per part)

e145bc455f1 5 hours ago | parent | prev [-]

what other tool do you recommend?

iberator 4 hours ago | parent [-]

just pay for storage instead. It's absurd that rich developers are doing ANYTHING but to pay for basic services - ruining the internet for those in real need.

we can't have nice things

wellf 5 hours ago | parent | prev [-]

Or.... backblaze B2

willis936 4 hours ago | parent | next [-]

Plus restic or borg or similar. I tried natively pushing from truenas for a while and it's just slow and unreliable (particularly when it comes to trying to bus out active datasets) and rsync encryption is janky. Restic is built for this kind of archival task. You'll never get hit with surprise bills for storing billions of small files.

encom 3 hours ago | parent | prev [-]

Have Backblaze software stopped being utterly awful, to the point of being almost nonfunctional, yet?

ziml77 an hour ago | parent [-]

What does Backblaze's backup software have to do with B2? Backblaze B2 is just storage that exposes the same API as S3. You can use any backup software that supports S3 as a target.