A nice source for this might be "The unbearable slowness of being: Why do we live at 10 bits/s?" [1].

A lot of people seem to think that training is limited to the 10 bits/s of processing mentioned here. But raw sensor input (mostly visual) is in the order of 21 quadrillion bits per year [2], according to the same paper. It seems likely to me that learning uses much more data than is ultimately expressed in behaviour.

[1] https://www.sciencedirect.com/science/article/pii/S089662732...

[2] Assuming 16 waking hours, and 10^9 bits/s of input that the paper mentions. 16*3600*365*10^9 = 21.024*10^15

That "raw sensor input" is massively, hideously spatially and temporally redundant.

Even the shallow encoding patterns of AV1 can easily crush a raw video stream down by a factor of 10^3 without losing either spatial coherence or semantic loading.

Humans live in self-similar environments and perform self-similar activities. Any practical training dataset has the "natural human data" beat on diversity by a crushing amount. And a system that can learn deeper patterns makes the sensor streams more redundant, not less.