Actually zstd internally splits data into frames, and frames can indicate the decompressed data size. So if we control the compressor we can make it so that all frames have the size information; it isn’t exactly seekable but at least it will not need to decompress the resource. https://python-zstandard.readthedocs.io/en/latest/concepts.h...

Given how fast zstd can decompress, this may or may not actually be a win: the time spent waiting for I/O might be so large that the decompression can fit within the wait time.

Which would have terrible performance for range requests starting late in a large file. For files that are frequently accessed that way, this could be prohibitive.

You could split the file into independently compressed blocks as well. But that'd reduce compression rate and require adding some kind of index for seeking.

Or they have an upper size limit for the file size they compress, since large files are rarely compressible text.

In any case it is something that needs the be handled before going live with a compressed cache. But the article sounds like they simply didn't implement compressed caching for those cases, which makes no sense.

Not with zstd, you could still support range requests. https://en.wikipedia.org/wiki/Zstd this whole subthread should take 10 minutes and glance over the spec and the capabilities. It would end a lot of wasted premature pontificating.

afaik, ZFS will read an entire Record at a time- and that's the same granularity as its compression.

It uses some form of keyframing. Not entirely free.