Remix.run Logo
duskwuff 2 hours ago

A common design in compressors is to use LZ as a first step, but to then represent the constant data and/or offset-length pairs from LZ using an entropy coder.

Deflate (as used in gzip) uses a Huffman coder. LZMA (as used by xz) uses a predictive range coder. Zstandard can use either Huffman or FSE. Some high-speed compressors like LZ4 skip the entropy coding stage entirely at the expense of compression ratio.

Bzip2 is an interesting aversion of this pattern - it uses the Burrows-Wheeler transform as a first pass instead of LZ. Unfortunately, this is one of the major reasons why it's so slow.