Back in my data hoarder days, I downloaded one of those torrents that had all the world's books in it. It was dunno how many terabytes, but way more than I had HDDs.

So I stripped out formatting, got rid of dupes, and tried out zstd, which was the hot new thing, along with the dictionary feature you describe, figuring it'd help. It didn't. I tried having one per book, one per multiple books, one for the whole archive.

It didn't work, or the gains were so marginal that I ended up scrapping the approach.

So it's not impossible that it can work, but stuff like regular json already compresses extremely well, I haven't found a scenario where it's a major boon.

Note that the dictionary options are only needed to improve compression ratios when compressing lots of small messages. If you have a bigger file (eg a tar file of Usenet messages) the regular Zstd compression will build a good dictionary without additional options.

As I understand there is no advantage in using a custom dictionary to compress 1 file. It benefits compressing _several_ (small) files.

The thing zstd got really right is fast decompression. For write-once read-never data like backups lzma (aka xz/7zip/lzip) is great. But it takes forever to decompress. On zstd I can get good compression while decompressing the file only marginally slower than reading the uncompressed file from SSD

Writing your files directly into a compressed stream and decompressing on the fly has become almost a standard workflow for any files I'm going to read and write sequentially anyways. No need for the data to ever exist uncompressed on the file system. Previous formats never did that for me because they either had too much overhead or too little gain, often both

How did I miss zstd?

Here are my benchmarks for 2.3 GB of jsonl, on a laptop. Compressed size, compress time, decompress time; using defaults.

    gzip  7.3%  21s  9s
    bzip2 4.6% 251s 50s
    bzip3 3.3%  82s 69s
    zstd  6.9%   2s  3s
    lzma  4.7%  51s  3s

Duckdb supports loading and saving to zstd for all it's base loading/saving formats csv/tsv/json/jsonlines, but, for good or bad, those are solid compression.

Under most r/w workloads, using parquet/lance/vortex/native-duckdb, with their built-in columnar compression will result in more performance AND space savings. Non-solid compression. Then, the query engine can push down your query predicate to a column row group level, instead of forcing it to decompress the entire dataset to operate.

Practical example: duckdb has syntax - https://duckdb.org/docs/lts/data/multiple_files/overview - to glob multiple files at once, but that really only works if you're applying push down query predicates instead of re-decompressing your entire data set per SELECT. I would say for most dataset, even 20%+ size is worth not having to decompress (or even download!) the entire dataset, to figure out if something fits the predicate.

After all, if you have to download and decompress the dataset back again to operate, then the "space savings" are gone.

Or lz4

all hail zstd, the one format to rule them all

Not really, it's a popular dictionary-based compression format.