> 8MB I believe for high levels

Yep, i found it in the source here:

- https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3...

Also, zstd docs say:

> Note: If windowLog is set to larger than 27, --long=windowLog or --memory=windowSize needs to be passed to the decompressor.

That always seemed annoying to me. They couldn't allocate 5 more bits somewhere to let the decompressor autodetect longer window sizes?

▲

dzaima 6 hours ago | parent | next [-]

> That always seemed annoying to me. They couldn't allocate 5 more bits somewhere to let the decompressor autodetect longer window sizes?

I believe this is just to prevent the decompressor from arbitrarily blowing up memory usage based on the input; I think if you want to accept long windows you can just always decompress with --long=63 regardless of whether the input needs it? (you will run out of RAM decompressing a long=63 file though of course)

▲

cesarb 4 hours ago | parent | prev | next [-]

> They couldn't allocate 5 more bits somewhere to let the decompressor autodetect longer window sizes?

It's actually 8 bits: https://www.rfc-editor.org/rfc/rfc8878.html#name-window-desc...

These command line parameters change the maximum the decompressor will allow. It's 128 MiB by default in the command line decompressor; other uses (like the "zstd" content coding for HTTP in web browsers) use a lower limit of 8 MiB (see https://www.rfc-editor.org/rfc/rfc9659.html).

▲

rmunn 5 hours ago | parent | prev [-]

2^27 is 128 megabytes. How much RAM do you want the decompressor to have to allocate for every file? Especially since you can't tell, by looking only at the file size of a compressed file, how many bytes it will decompress to. You could read the file header, but if it's a malicious "zip bomb" type of file, the header could be lying.

	▲	Dylan16807 an hour ago \| parent [-]
		If the header says the file is smaller than it really is, you've already allocated a small widow by the time you realize it lied, so it doesn't harm you here. If the header says the file is bigger than it really is, it can get you to allocate a pointlessly large window. But if a large allocation is the goal, they can make the file actually decompress that big without affecting the compressed size. So lying is pointless.