Remix.run Logo
▲ hn-aarvin 3 hours ago

I've been watching 3Blue1Brown's series on compression and intelligence^[1], which seems to discuss similar ideas.

In particular, his mention of an unknown floor for loss seems directly connected to Claude Shannon's entropy rate regarding compression—having to probe human brains to estimate the entropy of English at 1 bit per character, just as OpenAI doesn't know the exact entropy rate of its held-out codebase.

[1]: https://youtu.be/qbsb9DpdZ6s