Reminded me of pi filesystem (https://github.com/philipl/pifs), with enough digits of pi precalculated you might be able to do a decent compression program. The trick is in the amount of reasonable digits for that, if it’s smaller or bigger than that trained LLM.

▲

GuB-42 5 hours ago | parent | next [-]

I suspect that the length of the offset of your input data in pi is equal to the length of the input data itself, plus or minus a few bytes at most, regardless of the size of the input data.

That is: no compression, but it won't make things worse either.

Unless the input data is the digits of pi, obviously, or the result of some computation involving pi.

▲

MrLeap 3 hours ago | parent [-]

You could express the offset with scientific notation, tetration, and other big math number things. You probably don't need the whole offset number all at once!

	▲	GuB-42 2 hours ago \| parent [-]
		Actually, you do. You can use all the math stuff like scientific notation, tetration, etc... but it won't help you make things smaller. Math notation is a form of compression. 10^9 is 1000000000, compressed. But the offset into pi is effectively a random number, and you can't compress random numbers no matter what technique you use, including math notation. This can be formalized and mathematically proven. The only thing wrong here is that pi is not a random number, but unless you are dealing with circles, it looks a lot like it, so while unproven, I think it is a reasonable shortcut.

▲

sltkr 5 hours ago | parent | prev [-]

I'm going to be the nerd that points out that it has not been mathematically proven that pi contains every substring, so the pifs might not work even in theory (besides being utterly impractical, of course).

On a more serious note, as far as I understand these compression competitions require that static data is included in the size computation. So if you compress 1000 MB into 500 MB, but to decompress you need a 1 MB binary and a 100 MB initial dictionary, your score would be 500 + 100 + 1 = 601 MB, not 500 MB.

The relevance to this discussion is that the LLM weights would have to be included as static data, since the only way to regenerate them is from the initial training data, which is much larger than the resulting model. By comparison, pi based compression is the other way around: since pi is a natural constant, if your decompressor requires (say) a trillion digits of pi, you could write a relatively small program (a few kb) to generate them. It would be terribly slow, but it wouldn't affect your compression ratio much.

	▲	dataflow 13 minutes ago \| parent \| next [-]
		> I'm going to be the nerd that points out that it has not been mathematically proven that pi contains every substring Fascinating. Do you know if this has been proven about any interesting number (that wasn't explicitly constructed to make this true)?
	▲	netsharc 2 hours ago \| parent \| prev \| next [-]
		You mentioning the concept of pi containing every substring makes me think of Borges' Library of Babel. Ha, next: a compression algorithm that requires the user to first build an infinite library...
	▲	eru 4 hours ago \| parent \| prev \| next [-]
		> I'm going to be the nerd that points out that it has not been mathematically proven that pi contains every substring, so the pifs might not work even in theory (besides being utterly impractical, of course). Well, either your program 'works', or you will have discovered a major new insight about Pi. > On a more serious note, as far as I understand these compression competitions require that static data is included in the size computation. So if you compress 1000 MB into 500 MB, but to decompress you need a 1 MB binary and a 100 MB initial dictionary, your score would be 500 + 100 + 1 = 601 MB, not 500 MB. And that's the only way to do this fairly, if you are running a competition where you only have a single static corpus to compress. It would be more interesting and would make the results more useful, if the texts to be compressed would be drawn from a wide probability distribution, and then we scored people on eg the average length. Then you wouldn't necessarily need to include the size of the compressor and decompressor in the score. Of course, it would be utterly impractical to sample Gigabytes of new text each time you need to run the benchmark: humans are expensive writers. The only way this could work would be either to sample via an LLM, but that's somewhat circular and wouldn't measure what you actually want to measure in the benchmark, or you could try to keep the benchmark text secret, but that has its own problems.
	▲	charcircuit 5 hours ago \| parent \| prev [-]
		This only does 1 byte, so you only have to prove it contains the bits for 0-255.