Seems useful for s3 buckets. I suppose a script that watches for new objects, calculates its sha256 and stores that in a DB, then checks for duplicate hashes would be a fairly trivial task. Though s3 doesn’t support symbolic links so accounting would need to be handled by server side code.

▲

AceJohnny2 a day ago | parent | next [-]

While the idea of data-deduplication does apply everywhere, this specifically depends on the APFS CoW data feature, so it's not applicable outside of macOS (or APFS Volumes)

▲

rbanffy a day ago | parent [-]

The same concepts exist in ZFS, BtrFS and, IIRC, XFS.

▲

rf15 a day ago | parent [-]

also, worst case you can do this with link features? Which is basically every system under the sun.

▲

msh a day ago | parent [-]

but link is dangerous, as a edit made to one file will then be made to all copies.

For example, If I have two files /home/m/todo.txt and /home/m/backup2025/todo.txt that are identical and you then replace /home/m/todo.txt with a link to /home/m/backup2025/todo.txt and I later edit /home/m/todo.txt then backup copy would also be updated. With apfs clones the backup would not be modified when I updated the file.

	▲	rbanffy a day ago \| parent [-]
		Link might be desirable for your e-book or mp3 collections. Or .iso images, MAME ROMs, and other files that are unlikely to be changed.

▲

dmarinus a day ago | parent | prev [-]

I don't think it's very useful but there's already a checksum in the meta data of an S3 object: https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.h...

It might be better to keep an index of paths to checksums and use the checksum as object key in S3.

I'm not sure what you like to achieve but if you use restic with S3 as a backend you can achieve much better deduplicate and compression.