> deduplication at the file level: every file is now stored under its BLAKE3 hash

Blake3 is really a wonderfully fast cryptographic hash. I use it for my own "deduplication / integrity / berzerker" utility (which I made before LLMs were a thing).

If I've got a file named:

    DSC98731-b3-7b39197a22.JPG

then:

    - if that file doesn't checksum back to 7b39197a22 there's a file integrity problem (amazing and it already helped me troubleshoot issues)

    - if any other file has the same Blake3 7b39197a22 hash, it's a duplicate

    - if that 7b39197a22 checksum is in my database, "things can happen".

For example my DB can say "any file with a Blake3 hash of 7b39197a22 can always be deleted" or "any file with a Blake3 hash of 887463c09e, if it's got a generic filename like "dscXXXXX" can always be renamed to "20260722jackJohnAtTheBeach-b3-778463c09e.jpg" (or whatever suits you).

It's really great (and I know several here independently made similar schemes) and Blake3 is an amazing hash for those kind of use.

▲ Someone 5 hours ago | parent | next [-]

For those wondering like me: Blake3 generates hashes of at least 224 bits, not, as a literal reading of that comment indicates, 40 bits (which would be bad for file deduplication, giving you a 50% hash collision after around a million files)

	▲	TacticalCoder 2 hours ago \| parent [-]
		Yup sorry if I mislead people. I usually use 10 hexdigits (40 bits) but no matter how many there are, it's the x bits of the beginning of the checksum that are verified against the hash (in the examples I gave 40 bits). I wasn't very clear.

▲ maeln 4 hours ago | parent | prev | next [-]

Another cool thing about BLAKE3 is that it is a merkle tree. Not only it allows for good parallelism, but it also has a lot of cool property for data transfer. For example, you can check for partial validity, which allow for streaming error-detection and resend during the transfer. You only need to have the data and checksum in a way that you can start to reconstruct one or more subtree.

▲ gchamonlive 3 hours ago | parent | prev | next [-]

> I use it for my own "deduplication / integrity / berzerker" utility

Do you have it in a public repo you could share?

	▲	TacticalCoder 2 hours ago \| parent [-]
		No sadly I don't have any public repo: it's mostly really a collection of shell scripts and then the database one (for the berzerker/renamer) is in Clojure and I run it from my always-on REPL. But I know others did similar thing so maybe there are public repos out there. But in any case: it should now be the kind of thing relatively easy to vibe-code if it's for your own use.

▲ dist-epoch 5 hours ago | parent | prev [-]

It's annoying that most file formats don't checksum their own content.

Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage.

PostgreSQL, which prides itself by it's quality and reliability, only turned on checksums by default in the last version, 18.

This is one great benefit of using .zip files as file formats, you get this for free.

▲

SQLite an hour ago | parent | next [-]

Checksums use CPU cycles. SQLite will do checksums with an extension (https://sqlite.org/cksumvfs.html) but that is off by default since an overwhelming majority of developers are more interested in day-to-day performance than detecting (very rare) storage malfunctions.

▲

OskarS 4 hours ago | parent | prev | next [-]

I think it's reasonable for a DB like SQLite to delegate that to the filesystem. There is an overhead for doing it on the DB level, and since SQLite is just a file on the filesystem which, presumably, is serving many other files as well, why would you trust anything else on the filesystem if you don't trust SQLite? Like, your PHP script (or nginx server executable, or whatever) that is calling SQLite, that's not going to be check-summed either. Either you trust your filesystem or you don't, and if you don't, checksum and error correct on the filesystem level.

Though fair enough, it could offer it as an opt-in thing.

	▲	coldtea 3 hours ago \| parent [-]
		Because SQLite is a program, that will refuse to start or crash, and which you can trivially replace, if corrupted. Whereas your sqlite data are your data, and if they're corrupted they can be lost forever or propagate the issue to backups.

▲

TacticalCoder 2 hours ago | parent | prev [-]

> It's annoying that most file formats don't checksum their own content.

I agree.

> Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage.

I mostly run ext4 (desktop, laptops, etc.) but for my main server at home, it's a ZFS (mirrored) tank on an old server with ECC RAM.

And backups. So much backups.