Remix.run Logo
wyager 6 days ago

I have a project where I want two properties which are not inherently contradictory, but don't seem to be available together:

1. Huge compression window (like 100+MB, so "chunking" won't work)

2. Random seeking into compressed payload

Anyone know of any projects that can provide both of these at once?

rorosen 6 days ago | parent | next [-]

If seeking to frames is good enough for random seeking, this can be done with zeekstd already. The cli sets a custom window log (ZSTD_c_windowLog) on the compression context when creating binary patches[1], I regularly use it with a window size above 1G.

[1] https://github.com/rorosen/zeekstd/blob/main/cli/src/compres...

wyager 5 days ago | parent [-]

Nice, thanks.

wilted-iris 6 days ago | parent | prev [-]

That sounds like this with inline dictionaries added unless I’ve misunderstood you.