| ▲ | zhoBEENG 2 hours ago | |
Claude Shannon talks about this in A Mathematical Theory of Communication. He defines redundancy as one minus relative entropy, where relative entropy is the ratio of the language's actual average uncertainty per symbol to the maximum possible uncertainty if all alphabet symbols were completely random and equally likely. He gives some rather cute examples, like the language of Finnegans Wake by Joyce being very low redundancy (high efficiency in your words). He also states that crossword puzzles don't work in a perfectly efficient language, that 50% redundancy is pretty good for 2-d puzzles, and 33% redundancy good for 3-d puzzles. This has always been one of my favorite and in my mind most random corollaries in a paper. https://people.math.harvard.edu/~ctm/home/text/others/shanno... | ||