Remix.run Logo
Remix clone Hacker News
new | show | ask | jobsGithub
▲ klausa 7 hours ago

Exclusions are one thing, but I've had Backblaze _fail to restore a file_. I pay for unlimited history.

I contacted the support asking WTF, "oh the file got deleted at some point, sorry for that", and they offered me 3 months of credits.

I do not trust my Backblaze backups anymore.

▲orr721 4 hours ago | parent | next [-]

I had similar experience as well. They upgraded their client and server software something like 5 years ago which put forward different restrictions on character set used for password. I have used a special character which was no longer allowed. When I needed to restore files after disk failure I could not log in either in the app or on the website. The customer service was useless -- we are sorry, your fault. I have lost 1 TB of personal photos due to this as a paying customer. Never trust Backblaze.

▲nayhel89 5 hours ago | parent | prev | next [-]

I have the same experience with Backblaze. 3 years ago I tried to restore my files from Backblaze, using their desktop client.

First thing I noticed is that if it can't download a file due to network or some other problem then it just skips it. But you can force it to retry by modifying its job file which is just an SQLite DB. Also it stores and downloads files by splitting them into small chunks. It stores checksums of these chunks, but it doesn't store the complete checksum of the file, so judging by how badly the client is written I can't be sure that restored files are not corrupted after the stitching.

Then I found out that it can't download some files even after dozens of retries because it seems they are corrupted on Backblaze side.

But the most jarring issue for me is that it mangled all non-ascii filenames. They are stored as UTF-8 in the DB, but the client saves them as Windows-1252 or something. So I ended up with hundreds of gigabytes of files with names like фикац, and I can't just re-encode these names back, because some characters were dropped during the process.

I wanted to write a script that forces Backblaze Client to redownload files, logs all files that can't be restored, fixes the broken names and splits restored files back into chunks to validate their checksums against the SQLite DB, but it was too big of a task for me, so I just procrastinated for 3 years, while keeping paying monthly Backblaze fees because it's sad to let go of my data.

I wonder if they fixed their client since then.

▲willis936 6 hours ago | parent | prev | next [-]

Do you have any more details? This is a pretty big deal. The differentiators between Backblaze and Hetzner mostly boil down to this kind of thing supposedly not being possible.

▲klausa 5 hours ago | parent [-]

I’m on my phone so forgive the formatting, but here’s my entire support exchange:

- - -

Hey, I tried restoring a file from my backup — downloading it directly didn't work, and creating a restore with it also failed – I got an email telling me contract y'all about it.

Can you explain to me what happened here, and what can I do to get my file(s?) back?

- - -

Hi Jan,

Thanks for writing in!

I've reached out to our engineers regarding your restore, and I will get back to you as soon as I have an update. For now, I will keep the ticket open.

- - -

Hi Jan,

Regarding the file itself - it was deleted back in 2022, but unfortunately, the deletion never got recorded properly, which made it seem like the file still existed.

Thus, when you tried to restore it, the restoration failed, as the file doesn't actually exist anymore. In this case, it shouldn't have been shown in the first place.

For that, I do apologize. As compensation, we've granted you 3 monthly backup credits which will apply on your next renewal. Please let me know if you have any further questions.

- - -

That makes me even more confused to be honest - I’ve been paying for forever history since January 2022 according to my invoices?

Do you know how/when exactly it got deleted?

- - -

Hi Jan,

Unfortunately, we don't have that information available to us. Again, I do apologize.

- - -

I really don’t want to be rude, but that seems like a very serious issue to me and I’m not satisfied with that response.

If I’m paying for a forever backup, I expect it to be forever - and if some file got deleted even despite me paying for the “keep my file history forever” option, “oh whoops sorry our bad but we don’t have any more info” is really not a satisfactory answer.

I don’t hold it against _you_ personally, but I really need to know more about what happened here - if this file got randomly disappeared, how am I supposed to trust the reliability of anything else that’s supposed to be safely backed up?

- - -

Hi Jan,

I'll inquire with our engineers tomorrow when they're back in, and I'll update you as soon as I can. For now, I will keep the ticket open.

- - -

Appreciate that, thank you! It’s fine if the investigation takes longer, but I just want to get to the bottom of what happened here :)

- - -

Hi Jan,

Thanks for your patience.

According to our engineers and my management team:

With the way our program logs information, we don't have the specific information that explains exactly why the file was removed from the backup. Our more recent versions of the client, however, have vastly improved our consistency checks and introduced additional protections and audits to ensure complete reliability from an active backup.

Looking at your account, I do see that your backup is currently not active, so I recommend running the Backblaze installer over your current installation to repair it, and inherit your original backup state so that our updates can check your backup.

I do apologize, and I know it's not an ideal answer, but unfortunately, that is the extent of what we can tell you about what has happened.

- - -

I gave up escalating at this point and just decided these aren’t trusted anymore.

The files in question are four year old at this point so it’s hard for me conclusively state, so I guess there might be a perfect storm of that specific file being deleted because it was due to expire before upgraded to “keep history forever”, but I don’t think it’s super likely, and I absolutely would expect them to have telemetry about that in any case.

If anyone from Backblaze stumbles upon it and wants to escalate/reinvestigate, the support ID is #1181161.

▲notrealyme123 5 hours ago | parent | next [-]

Thank you for sharing this. A non-persistent backup service is on the same level as a zombie-insurance provider.

▲jeremyjh 3 hours ago | parent | prev | next [-]

This reminds me of the Seinfeld riff on car rental reservations. Anyone can make a backup. The important part is holding the backup. If Backblaze doesn’t always do that then it is practically worthless to everyone.

▲rincebrain 2 hours ago | parent | prev | next [-]

This seems absurd from a company offering backups as a service.

Especially if they allow them restoring all your data onto a drive and shipping it to you, they pretty clearly should have enough information available to them to test restorations of data, and the number of times I've heard that failure mode ("oh, we didn't track deletions well enough, so we only found out we deleted it when you tried restoring"), plus them saying they have made improvements to avoid this exact failure mode in newer client versions, makes me think they should have enough reports to investigate it.

...which makes me wonder if they did, and decided they would go bankrupt if they told people how much data they lost, so they decided to bet on people not trying restores on a lot of the lost data.

▲benjiro3000 5 hours ago | parent | prev [-]

[dead]

▲christoff12 6 hours ago | parent | prev [-]

wut