Imho deduplication belongs at the filesystem level, so the user won't (directly) see it or even know about it. Modern file systems like btrfs have the api for it.

▲

scheme271 a day ago | parent [-]

The dedup functionality in something like zfs or btrfs isn't all that great. It tends to be extremely memory hungry and to slow things down significantly. E.g. ZFS needs around 1-5GB of ram per TB of storage and writes need to be compared to a hash table to dedup properly.

Using hints or knowledge at the app level is a much better experience if the app can tell the FS that two files are identical. The FS doesn't have to worry about hashing blocks within the file, correcting alignments, etc.

▲

nagaiaida 21 minutes ago | parent | next [-]

> ZFS needs around 1-5GB of ram per TB of storage

...per unique TB written to datasets with deduplication enabled, of course, not the pool as a whole (even though that ram cost is indeed at the pool level).

there are plenty of useful things to be done with zfs dedupe at practically no cost if you design for it ahead of time. unfortunately for some reason people parachute into every thread where zfs comes up to say you can never ever turn it on, so people are taught not to even try.

▲

amelius a day ago | parent | prev [-]

Yes, but package managers are not that great either. Better keep them as simple as possible. And you don't have to do deduplication in an online fashion; you can do it overnight, if you want, as just a simple example.

▲

Timon3 20 hours ago | parent [-]

Do you specific issues with uv that makes you distrust the implementation? Otherwise that reasoning is pretty weird - it's possible to write good software, even when the existing options aren't good. Why would we ask them to limit themselves to what might make sense for worse code?

▲

amelius 12 hours ago | parent [-]

You're missing the point. No normal user program can ever do what a filesystem can do: deduplicate in a way that is hidden for the user of the filesystem. Unless you want to change everything into a black box managed by the package manager, making everything confusing for users and also maintainers.

▲

Timon3 9 hours ago | parent [-]

Why would the deduplication need to be hidden from the user? Why would everything need to become a blackbox?

Hardlinks have long been a well-documented feature of many file systems. I don't see how their use in a context users largely do not meddle with is too confusing or complicated.

▲

amelius 8 hours ago | parent [-]

Because why would you make everything more complicated if the file system can handle it just fine, plus you can automatically deduplicate other files that have nothing to do with package management?

	▲	Timon3 2 hours ago \| parent \| next [-]
		Because (going by scheme271's numbers) I might not want to waste lots of expensive RAM on something or mess around with offline deduplication, when a less resource-intensive alternative is entirely sufficient for whatever my use case is. Maybe it's because I've been recently playing around with deduplicating subtrees in a different domain, but I don't see what makes this feature too complex to be worth the cost. Sure, symlinks and hardlinks require some additional care compared to plain files, but it's easy enough to add tests for those cases.
	▲	ecnahc515 6 hours ago \| parent \| prev [-]
		Well uv runs on many systems and most don't have filesystem level dedupe configured, unfortunately. Anyways it can be both? It could let the filesystem handle it if it detected the FS has file or block dedupe and fallback to hardlinks otherwise. It doesn't but that could be done if it showed it was worth it.