Remix.run Logo
mmooss 7 hours ago

I've always wondered about similar designs: Doesn't calculating a hash of every block, on every read and every write, create lots of overhead? Why isn't that a problem?

Some systems have dedicated crypto co-processors for confidentiality (encryption) - e.g., I think drives with FDE, and I think Apple Silicon SoCs might have them. Can those be repurposed for hash calculation? What about systems that lack them?

chasil 7 hours ago | parent | next [-]

Both ZFS and modern btrfs support a large set of checksums.

Both implement sha256, which does impose a heavy speed penalty.

ZFS allows you to adjust the checksum on the fly, using something faster (Fletcher) if desired.

In btrfs, a global checksum is set at filesystem creation; xxhash is the best modern option.

There is a website: https://xxhash.com

Deduplication adds concerns for a strong hash free of collisions.

ThePowerOfFuet 5 hours ago | parent [-]

Fletcher has been the default for quite some time.

throw0101a 2 hours ago | parent [-]

Specifically "fletcher4":

* https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops...

* https://openzfs.github.io/openzfs-docs/Basic%20Concepts/Data...

* https://en.wikipedia.org/wiki/Fletcher%27s_checksum

* https://people.freebsd.org/~asomers/fletcher.pdf

* https://www.intel.com/content/www/us/en/developer/articles/t...

Originally documented in the paper "An Arithmetic Checksum for Serial Transmissions" (referenced also in RFC 1146, in the context of TCP).

yjftsjthsd-h 7 hours ago | parent | prev | next [-]

Yes, it adds some overhead, but it's fine IME. Granted, it helps that compression can significantly speed up performance. (I was very confused the first time I saw ZFS reading data faster than its drives were physically capable of, because it turned out the CPU could decompress faster than the drives could read)

crabbone 6 hours ago | parent | prev [-]

Programs like filesystems typically have their own schedulers that aggregate writes, it would be an extreme performance hit if they wrote each checksum individually, as a separate I/O operation. They are certainly bundled with some other data that needs to be written.

And if you are concerned about the compute rather than storage, then writing to a block device is still slow enough so that computing a checksum isn't important performance-wise.