> This is under-taught in comp sci so I'll say it in all caps for visibility. THERE IS NO KNOWN GENERIC WAY TO MEASURE INFORMATION ENTROPY!!!!

Both your comment and the OP seem to conflate a distribution with a sample from a distribution.

A distribution over strings has entropy. A single string can have a Kolmogorov complex but does not have entropy in the Shannon sense per se.

When you write a compression algorithm, you are often effectively building an algorithm to look at a single sample from an unknown distribution (the input data) and trying to invent a distribution that is both easy to describe and that produces the input data with respectably large probability. And then you output the description of the distribution and enough data to identify the sample in question, and the latter takes space that is roughly the entropy of the distribution you just made up.

> "What's the probability of this data assuming i always have the most perfect model to predict it?". You then calculate entropy based on that oracles answer.

Careful here. The most perfect model may be the one that outputs the data in question with probability one, and the entropy is exactly zero.

If you want information theory to give you meaningful answers, you need to ask it more carefully specified questions.

>A distribution over strings has entropy.

You have to limit yourself to each string being independent for that to be true which is a limitation that leads to the impossibility of calculating entropy generically.