I’m running an 8-drive ZFS RAIDZ2 pool. I’m wondering if you know — are the free space recommendations around ZFS cargo or real?

Like I’m already giving up two full drives for redundancy (which saved my ass - I recently had two drives fail on me in quick succession — both SSDs from what looks like an identical batch) but then the advice is kinda saying I need to keep at least another drive worth of space free for the pool to perform well and not crap itself. That hurts with current prices for sure.

▲

Confiks 12 hours ago | parent | next [-]

> I’m running an 8-drive ZFS RAIDZ2 pool. I’m wondering if you know — are the free space recommendations around ZFS cargo or real?

I'm not entirely sure, but it seems to me that free space (and the 20% reservation) is mostly a proxy for fragmentation, and you can therefore better look at fragmentation directly. That would mean that if you mostly store large files, there shouldn't be a lot of fragmentation even at high utilization. The whole "ZFS changes allocation algorithm from 80% usage on" is something of 10+ years distant past, and lots of things around the allocator have been improved. It's also something that probably isn't too different from the performance of other filesystems at high utilization, so it shouldn't be exaggerated.

	▲	Dylan16807 7 hours ago \| parent [-]
		> It's also something that probably isn't too different from the performance of other filesystems at high utilization, so it shouldn't be exaggerated. Well, other filesystems can defragment.

▲

gpt5 13 hours ago | parent | prev | next [-]

ZFS will auto-degrade performance if there isn't enough headroom. In addition, if you use SSD, you also want the headroom, because otherwise you end up writing and rewriting on the same small empty space which kills SSD (unless you have enterprise SSD, which have built-in headroom).

▲

gpt5 11 hours ago | parent [-]

Not sure why I'm being downvoted for being factually correct (and also, why I can't edit my above comment).

ZFS will definitely degrade write performance gradually from 90% utilization, and will hit a stronger cliff at 95%+. Same with SSD, using a consumer SSD above 90%+ utilization would rapidly degrade its lifetime. The effect will be smaller for very large pools and very large files, but the effect is stil there.

I'm repeating this so that people who set up these drives know what is actually going to happen.

▲

6581 10 hours ago | parent [-]

> Not sure why I'm being downvoted for being factually correct

Maybe because your statement about SSDs is incorrect. Consumer SSDs have spare blocks too, and wear leveling prevents the scenario you're describing.

▲

gpt5 9 hours ago | parent [-]

I appreciate the reply, but my warning consumer SSD is not incorrect. Perhaps I should have clarified that While they typically have a small overprovisioning of 7%, enterprise SSDs have close to 4 times as much, exactly to allow more writes (which is very relevant if running a ZFS raid on the drive). Keeping 10-20% headroom will prolong your SSD's life significantly, and increase performance of ZFS.

	▲	seniorThrowaway 3 hours ago \| parent [-]
		Go ahead and let them try consumer SSD's. It seems to be a lesson everyone has to learn with ZFS once.

▲

cm2187 10 hours ago | parent | prev | next [-]

For home usage, if you have backups raidz1 is fine (just do an incremental backup at the first sign of trouble). If you don’t have backups, then you probably shouldn’t be running a NAS in the first place.

▲

bensyverson 4 hours ago | parent | next [-]

The risk of z1 is that if you get a read error during resilvering, that data is permanently corrupted. The odds of this happening go way up the larger your individual drives are. This is why I chose RAIDZ2 for my NAS. I've had to resilver 2-3 times over the past 10 years, and never lost a byte of data.

▲

ppchain 5 hours ago | parent | prev | next [-]

In the Linux ISO community the math is different because none of the data is essential, however there is a lot of it. Backing it all up even one time to an off-site location would be a 2x expense for a hobby. So 3-2-1 is hard to justify.

Instead the math can be different e.g. it's ok to lose some data but not all. Therefore you might prefer the unraid approach over zfs where losing more than parity doesn't kill the whole pool.

	▲	dwedge 4 hours ago \| parent [-]
		I'm actually in this boat right now. I have 2x26TB drives in a 4 base nas (two empty, plus 3x M2 empty) and I'm trying to decide the best way to set them up. I have around 3TB of backups that I care about as backups (but they don't need to be online, just archive backups at this point), and around 5TB of media that can be replaced. I don't want to lose 26TB just for a mirror, and I have a spare 8TB USB HDD. I'm torn between unraid and just Debian, and I'm torn between just two separate devices and one RAID 1 partition one RAID 0

▲

bluGill 6 hours ago | parent | prev | next [-]

For most people snapshots are enough backup. It is still useful to have an offsite backup, but realistically fire is rare enough that you can risk it, and that is about the only risk most people have (you can have your NAS in a location likely far from where a fire might break out to reduce the risk farther).

▲

cm2187 5 hours ago | parent [-]

Fire, malware, accidental deletions, capricious RAID controller (pre-ZFS). And that’s only the stuff that happened to me. Add power surge, theft, correlation in SSD failures (eg power on counter overflow firmware bug), damaging the array while moving, etc.

▲

bluGill 4 hours ago | parent [-]

Malware, accidental deletion, RAID controller issues - all things ZFS with snapshots are immune to. You can decide if the other worry you or not - the risk is not zero, but it may be acceptable. Your backups are all subject to similar issues.

	▲	cm2187 4 hours ago \| parent [-]
		Snapshots don’t protect you from malware. I bet 99% of home users use the same credentials on all their machines, once a malware compromised one, the others are compromised within seconds. [edit] also snapshots aren’t really workable for large files. Remux a movie file and now it occupies twice the space.

▲

Cyph0n 6 hours ago | parent | prev | next [-]

The middle ground I go with is to structure the pool using datasets and zvols, and then periodically backup the critical ones using zfs send. For example, I can live without my movie collection, but I cannot afford to lose personal photos.

▲

layer8 7 hours ago | parent | prev [-]

This isn’t black and white. You might have huge amounts of non-essential data that aren’t worth the cost of off-site backups, but worth the cost of an extra disk of redundancy to lessen the risk. Even when you do have backups, it will reduce the risk of extended downtime (and possibly egress costs) caused by having to restore large amounts of data from backups.

▲

vladvasiliu 11 hours ago | parent | prev [-]

I don't think they are. I've already filled a ZFS pool almost to the brim with no adverse effects.

	▲	barrkel 10 hours ago \| parent [-]
		My experience has been escalating fragmentation past 90%, along with lower performance owing to the fragmentation.