This is under-taught in comp sci so I'll say it in all caps for visibility. THERE IS NO KNOWN GENERIC WAY TO MEASURE INFORMATION ENTROPY!!!!

Seriously. There isn't. You might say 'but measure the number of 1's vs 0's' and i'll just reply with '101010101010 repeated'. So you up it and start saying ok maybe measure frequencies of pairwise bits then and i just reply by cycling 00,01,10,11 endlessly etc. Frequency counts and any other measure you can think of doesn't actually work.

The root definition will state it's just based on the properties of the prediction but... what model do you use for the prediction? This gets to the heart of the matter;

The measure of information entropy itself is actually equivalent to creating on omniscient oracle. It requires being able to ask the question "Oh great oracle, what's the shortest program that can reproduce X"? or equivalently "What's the probability of this data assuming i always have the most perfect model to predict it?". You then calculate entropy based on that oracles answer.

Kolmorogorov, AI and information entropy are all pointing to the same thing. The reason people get confused by the fact that the frequencies of base 10 digits is the same for digits of pi and a truly random sequence is just because the counts of base 10 digits isn't a measure of entropy at all. Measuring entropy literally requires an omniscient oracle.

I think the explanation of entropy's blind spot is a bit off. It's not actually a problem for entropy if something is generated by a rule, you can calculate entropy for things like the continuous fractions for instance, with an easy rule to generate them for any particular number. Likewise for decimal expansions.

The real blind spot is that entropy is meaningless for a specific sequence, you can't really ask about the entropy of pi if you don't have a theory for how the numbers are generated. Sure if it is pick a uniformly random real number between 0 and 10 then both files have equivalent entropy, but sending pi is also vanishingly unlikely.

There's actually a more subtle way in which this is a blind spot, which takes a bit more machinery. You can define entropy for an ergodic system, which could be considered a kind of mathematical RNG. Now as it turns out this provides a way to generate something almost equivalent to a particular distribution except that this argument only holds for most starting points not all. A direct example would be how pi generates a perfectly fine random distribution of digits (we think) but something like 1/3 does not.

I don’t understand the claim that Kolmogorov lower bounds are uncomputable. Three digit strings are unlikely to run as a program so surely 3 is a lower bound on Kolmogorov of the first million digits of pi, no? To tighten bound, one would need to try out every 4-digit program, every 5-digit program, and so on. Either these produce the first million digits of pi or they don’t when the program is run, so each possibility is checkable in finite time, no? There must be something I’m missing here.

I've been down this rabbit hole.

I tried to make a codebook of everything. You can get surprisingly good compression by assigning words and phrases to numbers.

My particular attempt had phrases up to 12 words.

No grammar or anything...to test the program, when I had a random thought, I tested to see if I could encode that thought. I could and on average the thought/message was compressed by 2. There was very few times where I wasent able to encode the idea into the available building blocks or language model of the database.

https://i.postimg.cc/WpBcB4HK/IMG-20260510-103853.jpg

Lately I’ve felt Kolmogorov complexity is an unfair measurement because it takes for granted your underlying programming language as treats it as zero cost. In theory you could create a custom language and embed the program as data and “compress” a large random sequence with a better Kolmogorov complexity for that specific language than Pi, simply by not exposing the ability in the language to even work with Pi. I think what’s maybe more interesting is when you take into account the work of Dr. Futamura and the idea of Jones Optimality and view things through that lens.

The choice of pi as a number whose sequence of digits is random is a bit of a weak point of the argument in the post. It is not even known whether every digit 0-9 appears infinitely many times in pi (a weak version of the normal number conjecture). So we do not really know (but strongly believe) that the sequence of pi digits would truly appear random. This of course does not detract from beautiful arguments and the general ideas in this post.

The thing that frustrates me about this argument is that there is no shortest program that produces pi. You need a computer to run it, which is massive non compressed data, or a human to calculate stuff, an uncountable amount of entropy.

I see that the irrational pi has a smooth distribution of digits and a file full of zeroes is compressible, but they are both sort of magically part of a world that does not run programs and thus not quite different in a practical sense.

Just my thoughts and sorry for the confusion.

there are many instances of claude in here, so not sure what that disclaimer was about.