| ▲ | ACCount39 an hour ago | |
Why not? If our objective is, for example, "compress the sum of all of humankind's knowledge", then we have a bit of a conundrum: that objective contains human knowledge of math, but also of religion, and of soap opera plots. You, a human, could easily make an argument that one of those things is more important than the others, and thus should dominate. But an optimal compressor does not care. All of those things are in its objective. It weights them accordingly. It will compress both math and soap opera into its "universal information state" - and if it finds patterns that happen to help compress both, then all the better for it. As such, if we take "compressor" as LLM and "objective" as its training datasets, a capable LLM would be able to both solve math problems and give a preacher-like sermon. But it would not let its knowledge of religion intrude into its performance in math, or the other way around, to an unwarranted level. The compressor lives by "non-overlapping magisteria." > You can write a script in an endless amount of programming languages, you can segment the task into an almost innumerable set of sub steps (for example different scripts for different steps in the process), so what exactly does compression of that information mean when it can be compressed and predicted in so many different ways? It means that, instead of storing every possible permutation, we get a compressed representation that decomposes - separates "segment a video" from "write code in language A" or "use tool B". And then those elements can be recombined into the near-infinite permutations to decompress. The compressor finds the commonalities and exploits them to fit "infinite" into "finite" better. | ||