The benchmarks are disingenuous, to the point of looking cherry-picked. The block size for bzip3 is set to 512MB, but the window size for zstd is left to its default (8MB I believe for high levels). So in this corpus, which is made up of all versions of Perl source code concatenated, the window is too small to see all the identical files and just match them. Also corpora made out of very long repetitions are pretty much the best case scenario for BWT-based compressors.

If we match the window size of zstd to that of bzip3 we get dramatically different results:

    % gzcat *.gz | time zstd -T8 -16 | wc -c  # baseline
     2819113884
    zstd -T8 -16  2054.50s user 3.47s system 783% cpu 4:22.80 total

    % gzcat *.gz | time zstd -T8 -16 --long=29 | wc -c
     196405076
    zstd -T8 -16 --long=29  1083.06s user 2.41s system 783% cpu 2:18.55 total

Almost 15x smaller than the baseline, and more than 2x smaller than bzip3, also CPU time halves (since long matches are found earlier, so there's less work to do).

(the baseline number is slightly different because I don't have the exact Perl version set used by the author)

Also, in the benchmarks using lrzip, which would make the window size less relevant, zstd is not even compared.

▲ nvme0n1p1 6 hours ago | parent | next [-]

> 8MB I believe for high levels

Yep, i found it in the source here:

- https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3...

Also, zstd docs say:

> Note: If windowLog is set to larger than 27, --long=windowLog or --memory=windowSize needs to be passed to the decompressor.

That always seemed annoying to me. They couldn't allocate 5 more bits somewhere to let the decompressor autodetect longer window sizes?

▲

dzaima 6 hours ago | parent | next [-]

> That always seemed annoying to me. They couldn't allocate 5 more bits somewhere to let the decompressor autodetect longer window sizes?

I believe this is just to prevent the decompressor from arbitrarily blowing up memory usage based on the input; I think if you want to accept long windows you can just always decompress with --long=63 regardless of whether the input needs it? (you will run out of RAM decompressing a long=63 file though of course)

▲

cesarb 4 hours ago | parent | prev | next [-]

> They couldn't allocate 5 more bits somewhere to let the decompressor autodetect longer window sizes?

It's actually 8 bits: https://www.rfc-editor.org/rfc/rfc8878.html#name-window-desc...

These command line parameters change the maximum the decompressor will allow. It's 128 MiB by default in the command line decompressor; other uses (like the "zstd" content coding for HTTP in web browsers) use a lower limit of 8 MiB (see https://www.rfc-editor.org/rfc/rfc9659.html).

▲

rmunn 5 hours ago | parent | prev [-]

2^27 is 128 megabytes. How much RAM do you want the decompressor to have to allocate for every file? Especially since you can't tell, by looking only at the file size of a compressed file, how many bytes it will decompress to. You could read the file header, but if it's a malicious "zip bomb" type of file, the header could be lying.

	▲	Dylan16807 an hour ago \| parent [-]
		If the header says the file is smaller than it really is, you've already allocated a small widow by the time you realize it lied, so it doesn't harm you here. If the header says the file is bigger than it really is, it can get you to allocate a pointlessly large window. But if a large allocation is the goal, they can make the file actually decompress that big without affecting the compressed size. So lying is pointless.

▲ cb321 5 hours ago | parent | prev | next [-]

I just tried a tar file of a git clone of the linux kernel sources where the most recent commit is 72c395024dac5e215136cbff793455f065603b06 (early Feb of this year). zstd -19 got a slightly smaller size (3582930348 bytes vs bz3 -b 511's 3597411687 bytes or 0.4% advantage to zstd). More significantly 4-core zstd decompression was 2.05 seconds vs a whopping 297 seconds for bzip3 -dj4 - 145x or over 2 orders of magnitude slower (about as much time to decode as to encode in the first place). bzip3 1.5.3 compiled with gcc-16.1.0. Granted, the .git objects are all compressed already and uncompressed tar-ball was only 5426667520 bytes, but even so...A lot of people care about fast(-ish) decompression. Maybe I did something wrong? Maybe `rm -rf .git` first would be a better benchmark?

▲ optionalsquid 6 hours ago | parent | prev | next [-]

How does memory usage compare between your two runs?

The benchmarks report 687M for their run of zstd, and 12178M and 18301M for the two runs of BZip3. Which itself is a bit eyebrow raising

▲ xxs 5 hours ago | parent | prev | next [-]

Nice catch.

it has been a long time since: "lies, damn lies, benchmarks" failed to hold true. Sometimes I wonder why gaming benchmarks has become so common.

▲ pajko 5 hours ago | parent | prev | next [-]

This was my point too, but related to LZMA(2?) / xz, as the exact parameters were not specified while it supports setting compression level up to -9e, and the dictionary size can be controlled directly as well (in addition to quite a lot of fine-tuning knobs), increasing it up to 1536 MiB.

▲ CJefferson 7 hours ago | parent | prev | next [-]

Wow, that is widely disingenuous, I don't really think there is any excuse for that, I don't believe someone deep in compression algorithms wouldn't know they could adjust the block size, and 512GB is a huge block size for bzip3, as it needs to basically all be in memory so you can't pretend that's just 'the standard value'.

	▲	ot 6 hours ago \| parent \| next [-]
		> 512GB is a huge block size for bzip3 Sorry! That was a typo, it should have been 512MB (now fixed). Still huge.
	▲	6 hours ago \| parent \| prev [-]
		[deleted]

▲ sltkr 4 hours ago | parent | prev [-]

Can you tell me what the zstd invocation is that corresponds to the default invocation of bzip3, which uses block size 16 MiB (according to the man page)?

I got some really good results with bzip3 compression Wikipedia XML dumps, and I would like to check if it's actually better or if I was just calling zstd wrong.

	▲	loeg 4 hours ago \| parent [-]
		If you want a 16MB window, use `--long=24` (2^24 is 16M). (I believe this is larger than the default window for zstd level 3, but smaller than the default window at higher compression levels.)