| ▲ | mmooss 7 hours ago | ||||||||||||||||
I've always wondered about similar designs: Doesn't calculating a hash of every block, on every read and every write, create lots of overhead? Why isn't that a problem? Some systems have dedicated crypto co-processors for confidentiality (encryption) - e.g., I think drives with FDE, and I think Apple Silicon SoCs might have them. Can those be repurposed for hash calculation? What about systems that lack them? | |||||||||||||||||
| ▲ | chasil 7 hours ago | parent | next [-] | ||||||||||||||||
Both ZFS and modern btrfs support a large set of checksums. Both implement sha256, which does impose a heavy speed penalty. ZFS allows you to adjust the checksum on the fly, using something faster (Fletcher) if desired. In btrfs, a global checksum is set at filesystem creation; xxhash is the best modern option. There is a website: https://xxhash.com Deduplication adds concerns for a strong hash free of collisions. | |||||||||||||||||
| |||||||||||||||||
| ▲ | yjftsjthsd-h 7 hours ago | parent | prev | next [-] | ||||||||||||||||
Yes, it adds some overhead, but it's fine IME. Granted, it helps that compression can significantly speed up performance. (I was very confused the first time I saw ZFS reading data faster than its drives were physically capable of, because it turned out the CPU could decompress faster than the drives could read) | |||||||||||||||||
| ▲ | crabbone 6 hours ago | parent | prev [-] | ||||||||||||||||
Programs like filesystems typically have their own schedulers that aggregate writes, it would be an extreme performance hit if they wrote each checksum individually, as a separate I/O operation. They are certainly bundled with some other data that needs to be written. And if you are concerned about the compute rather than storage, then writing to a block device is still slow enough so that computing a checksum isn't important performance-wise. | |||||||||||||||||