> “bzip3 is not yet listed on the large text compression benchmark” It is now

And it comes in relatively well, in my opinion. I'm a compression amateur, but bzip3 is the first entry I recognize as a general purpose compression program.

> “Hi, tool author here.” A useful explanation of Burrows-Wheelers transform as used by bzip3

FWIW, the Burrows-Wheelers transform is also used by bzip2, so this isn’t a new feature even though that quote kinda sounds like it is.

For that type of structured data (logs and such), a custom dictionary can be extremely effective. Zstd among others support generating a custom dictionary. You just run zstd --train over the data first, and then feed that in when you run zstd. For example: I found ~10 gigabytes of Usenet headers compress to ~700 MB using Zstd and 1 MB shared dictionary -- and that's with each header individually compressed, so o(1) lookup time.

zstd is the go-to compression format these days. It's even supported in low-level software such as many linux filesystems.

I don't know much about duckdb but it looks like it supports zstd too: https://duckdb.org/docs/lts/data/json/loading_json

For structured logs and json I've found a lot of success with PPM-style schemes.

If your JSON file has many of the same object, you could see ratios in the single digits.

I'd recommend trying openzl for jsonl.

> 8MB I believe for high levels

Yep, i found it in the source here:

- https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3...

- https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3...

Also, zstd docs say:

> Note: If windowLog is set to larger than 27, --long=windowLog or --memory=windowSize needs to be passed to the decompressor.

That always seemed annoying to me. They couldn't allocate 5 more bits somewhere to let the decompressor autodetect longer window sizes?

I just tried a tar file of a git clone of the linux kernel sources where the most recent commit is 72c395024dac5e215136cbff793455f065603b06 (early Feb of this year). zstd -19 got a slightly smaller size (3582930348 bytes vs bz3 -b 511's 3597411687 bytes or 0.4% advantage to zstd). More significantly 4-core zstd decompression was 2.05 seconds vs a whopping 297 seconds for bzip3 -dj4 - 145x or over 2 orders of magnitude slower (about as much time to decode as to encode in the first place). bzip3 1.5.3 compiled with gcc-16.1.0. Granted, the .git objects are all compressed already and uncompressed tar-ball was only 5426667520 bytes, but even so...A lot of people care about fast(-ish) decompression. Maybe I did something wrong? Maybe `rm -rf .git` first would be a better benchmark?

How does memory usage compare between your two runs?

The benchmarks report 687M for their run of zstd, and 12178M and 18301M for the two runs of BZip3. Which itself is a bit eyebrow raising

Nice catch.

it has been a long time since: "lies, damn lies, benchmarks" failed to hold true. Sometimes I wonder why gaming benchmarks has become so common.

This was my point too, but related to LZMA(2?) / xz, as the exact parameters were not specified while it supports setting compression level up to -9e, and the dictionary size can be controlled directly as well (in addition to quite a lot of fine-tuning knobs), increasing it up to 1536 MiB.

Wow, that is widely disingenuous, I don't really think there is any excuse for that, I don't believe someone deep in compression algorithms wouldn't know they could adjust the block size, and 512GB is a huge block size for bzip3, as it needs to basically all be in memory so you can't pretend that's just 'the standard value'.

Can you tell me what the zstd invocation is that corresponds to the default invocation of bzip3, which uses block size 16 MiB (according to the man page)?

I got some really good results with bzip3 compression Wikipedia XML dumps, and I would like to check if it's actually better or if I was just calling zstd wrong.

“Additional benchmarks on the same dataset” (2025) explores various compression levels with and without long range mode: https://news.ycombinator.com/item?id=42901476

It's beyond me why such foundational libraries don't have formal correctness proofs attached these days.

That's like comparing apples to pears. OpenZL is not general purpose. You specify a format for data and it compresses that format. Specifying a general "could be anything" format would be interesting, but I doubt it would compress as well.

The author of bzip3 probably doesnt like pushover licenses so made user rights stronger

bzip3 is not made by the same author as bzip2, that's why there is a licence disparity.

Please stop questioning people's choice of license.

If you don't like it, you're entitled to not using it.

Tool author here. bzip2 also has this clause, in fact it has been lifted from its dist tarball README verbatim. So does lzma, xz, or in practice any open source program that you use.

Is that all that different than the standard MIT License:

> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE....

This big warning gave me pause, too. Why not have a compression option that automatically checks the data after compression (by decompressing and checking against a hash of the original data), which would reduce the probability of undetected errors to that of a hash collision? i.e. like `7z a` followed by `7z t`, but in one command.

So apparently zamadatix's point wasn't well taken, but I think it could be an effective approach to store the same data twice over to increase the likelihood of successful recovery. Could be even be even ten times, or e.g. 6+4 with redundant recovery coding.

Although you do have a point that the compressed data might be more difficult to decipher, if it doesn't have sufficient redundancy to skip bad parts, or if it is essential that the data is aligned in a certain way (e.g. disk images, and probably many other formats) and the format doesn't take this into account. Shorter window sizes, window reset markers, and explicit offset information could mitigate those problems.

Depends on the specific scenario. E.g. efficient compression can speed up recovery from slow media and provide space for increased parity.

Those tutorials are about as old as the author of the repository.

Depends - if parallel decompression of any bzip3 archive is possible I'd consider it fair, if it requires special flags on archive creation I'd consider it unfair. I didn't see any description about that on that page.

pbzip2 only can do parallel decompression on archives created with pbzip2, otherwise it'll fall back to single thread. There nowadays seems to be lbzip2, though, which claims to be able to add SMP support for standard bzip2 archives.

I'll need to try that next time I'm working with large archives - I learned about the pbzip2 limitations the hard way last time I was shuffling around a few multi-10GB archives, and was trying to speed things up fully utilising my 32 core threadripper.

“for fairness, the benchmarks have been performed using single thread mode” (2025) https://news.ycombinator.com/item?id=42902241

No. It is an unrelated algorithm.

You're probably thinking of LZMA2.

I have not experimented with bzip3 recently, but more than a year ago I have done many tests with it.

Initially I was extremely impressed with it, because in a lot of tests it succeeded to compress hard-to-compress files, like movies, and in many cases it demonstrated a much better compromise between speed and compression ratio than zstd, i.e. depending on the command parameters I could make it either compress better than zstd at similar compression/decompression speed, or compress/decompress faster at a similar compression ratio.

Alas, the initial extremely favorable conclusion was short-lived, because trying later bzip3 on other data files gave worse results than zstd.

So the final conclusion was that the performance of bzip3 was somewhat unpredictable, being highly data dependent. For some files it provided outstanding compression ratio or speed, but for others it was inferior.

The problem was that without doing a compression there was no way to guess whether a file would be among those preferred by bzip3 or by zstd or by xz.

So now I would use it only for a file for which I want maximum compression and which I would compress once and decompress many times, so I can afford a very long compression time, during which I would test multiple compression algorithms, including bzip3 and zstd, with multiple parameter choices, and I would eventually choose the one that offers the best compromise between compression ratio and decompression time, for that particular file.

It certainly is a competitive compression algorithm, but unless it has changed since I last tested it, you cannot guess for which files it would win the compression competition.

with zstd at level 16 with default params (dict size, ...). Serious compression starts at level 19 and with much higher dict sizes.

how is this an honest benchmark:

    bzip3 ... 12178M memory
    zstd  ...   687M memory

The benchmark is very rudimentary. It does not test different levels/settings apart from its own -b 256/512 (does it affect decompression?), it doesn't measure compression time and memory usage. It does not specify parallel vs single-threaded (it mentions parallel on the one decoding number but what about the others?).

The lrzip test is interesting but it omits for example zstd and doesn't even have (de-)compression timings.

A lot more numbers are needed to present a fair and informative comparison.

I don't want this to be a swipe against bzip3, I only want to point out the presented benchmarks could be a lot better.