It's funny but I'm on HN so I can't resist pointing out the joke doesn't math TFA, their argument is that the underlying internet distribution is trained away, not retained.

Maybe the real underlying distribution IS a lot of text from people just spewing out feel good words for socialising. I think that might be side effect of the fact that meaningful text is harder to make even for humans, so there is smaller quantity of meaningful text.