Okay now this is weird.

I can reproduce it just fine ... but only when compressing all PDFs simultaneously.

To utilize all cores, I ran:

    $ for x in *.pdf; do zstd <"$x" >"$x.zst" --ultra -22 & done; wait

I ran this again and it produced the same 2M file from the source 1.1M file. However when I run without paralellization:

    $ for x in *.pdf; do zstd <"$x" >"$x.zst" --ultra -22; done

What's going on here? Surely '-22' disables any adaptive compression stuff based on system resource availability and just uses compression level 22?

I'm not sold on the idea of adding compression to PDF at all, I'm not convinced that the space savings are worth breaking compatibility with older readers. Especially when you consider that you can just compress it in transit with e.g HTTP's 'Content-Encoding' without any special PDF reader support. (You can even use 'Content-Encoding: br' for brotli!)

If you do wanna change PDF backwards-incompatibly, I don't think there's a significant advantage to choosing gzip to be honest, both brotli and zstd are pretty widely available these days and should be fairly easy to vendor. But yeah, it's a slight advantage I guess. Though I would expect that there are other PDF data sets where brotli has a larger advantage compared to gzip.

But what I really don't get is all the calls to use zstd instead of brotli and treating the choise to use brotli instead of zstd as some form of Google conspiracy. (Is Facebook really better?)

I just did some interactive shell loops and globs to compress everything and output CSV which I processed into an ASCII table, so I don't exactly have a pipeline I can modify and re-run the tests with compression speeds added ... but I can run some more interactive shell-glob-and-loop-based analysis to give you decompression speeds:

    ~/tmp/pdfbench $ hyperfine --warmup 2 \
    'for x in zst/*; do zstd -d >/dev/null <"$x"; done' \
    'for x in gz/*; do gzip -d >/dev/null <"$x"; done' \
    'for x in xz/*; do xz -d >/dev/null <"$x"; done' \
    'for x in br/*; do brotli -d >/dev/null <"$x"; done'
    Benchmark 1: for x in zst/*; do zstd -d >/dev/null <"$x"; done
      Time (mean ± σ):     164.6 ms ±   1.3 ms    [User: 83.6 ms, System: 72.4 ms]
      Range (min … max):   162.0 ms … 166.9 ms    17 runs
    
    Benchmark 2: for x in gz/*; do gzip -d >/dev/null <"$x"; done
      Time (mean ± σ):     143.0 ms ±   1.0 ms    [User: 87.6 ms, System: 43.6 ms]
      Range (min … max):   141.4 ms … 145.6 ms    20 runs
    
    Benchmark 3: for x in xz/*; do xz -d >/dev/null <"$x"; done
      Time (mean ± σ):     981.7 ms ±   1.6 ms    [User: 891.5 ms, System: 93.0 ms]
      Range (min … max):   978.7 ms … 984.3 ms    10 runs
    
    Benchmark 4: for x in br/*; do brotli -d >/dev/null <"$x"; done
      Time (mean ± σ):     254.5 ms ±   2.5 ms    [User: 172.9 ms, System: 67.4 ms]
      Range (min … max):   252.3 ms … 260.5 ms    11 runs
    
    Summary
      for x in gz/*; do gzip -d >/dev/null <"$x"; done ran
        1.15 ± 0.01 times faster than for x in zst/*; do zstd -d >/dev/null <"$x"; done
        1.78 ± 0.02 times faster than for x in br/*; do brotli -d >/dev/null <"$x"; done
        6.87 ± 0.05 times faster than for x in xz/*; do xz -d >/dev/null <"$x"; done

    +-------+-------+--------+-------+
    | gzip  | zstd  | brotli | xz    |
    +-------+-------+--------+-------+
    | 143ms | 165ms | 255ms  | 982ms |
    +-------+-------+--------+-------+

They don’t seem to provide a detailed comparison showing how each compression scheme fared at every task, but they do list (some of) their criteria and say they found Brotli the best of the bunch. I can’t tell if that’s a sensible conclusion or not, though. Maybe Brotli did better on code size or memory use?

Hey, they did all the work and more, trust them!!!

> Experts in the PDF Association’s PDF TWG undertook theoretical and experimental analysis of these schemes, reviewing decompression speed, compression speed, compression ratio achieved, memory usage, code size, standardisation, IP, interoperability, prototyping, sample file creation, and other due diligence tasks.