Noting the theory papers on Kolmorogov complexity. For those not familiar, Ilya argues that the reason why neural networks generalize -- why they work at all -- is because they are effectively finding a simple description of their training data, converging down onto the limit of the Kolmorogov complexity. [1]

[1] https://www.youtube.com/watch?v=AKMuA_TVz3A

That's true of all statistical models, it's not some magic property of neural networks.