The output of a GPT is an interpolation (an estimation of new data points inside the range of known data) rather than extrapolation (estimations outside that range).

99% of the time we don't need a true intellectual breakthrough to get the job done, and often 'new ideas' are simply riffs on or blends of old ones, like fashion or music genres.

The worry to me, however, is that if society comes to rely on this form of 'AI' then eventually the model collapse bleeds into academia (e.g. grant proposals reviewed by AI?) causing a kind of incremental sociocognitive atrophy. Everything becomes a reaffirmation of the status quo.

That being said I think people said something similar about electronic calculators (that if you couldn't do long division by hand then you'd be too incompetent for higher-level calculus.)

The more I read these think-pieces on AI the clearer it is to see that they have very little to do with LLM technology, and are really just talking about the collapse of the set of values constructed during the enlightenment. The sociocognitive atrophy that you describe is inherent to the idea that newness and progress are the ultimate goal that we should organize society around. The problem is that newness and progress can only be defined relative to a status-quo, and hence are incoherent goals in themselves. When pursued far enough, they become their own strange, monstrous status-quo that betrays the original intentions of the people who pursued these values. Hence "we mistake the flattening for progress". This was the case before AI, it's just that AI makes it much harder to ignore, and in many ways encapsulates the problem.

The point w/ electronic calculators is the same point made by Plato regarding books. It used to be easy to laugh off these concerns, not so much today. Imo, this is the real progress: people are now asking meaty questions regarding the ultimate human purpose of books, calculators and technology.

> if you couldn't do long division by hand

But the people studying math and the related fields are able to do division by hand on paper. They are just slow when doing it.

I believe that the calculator was meant to solve the slowness problem rather than eliminate the need to fundamentally understand division.

>The output of a GPT is an interpolation (an estimation of new data points inside the range of known data) rather than extrapolation (estimations outside that range).

That's a common meme but it's the opposite of true. Everything big models, not just transformers, mathematically do is extrapolation in the feature space, almost never interpolation. They're perfectly able of combining the ideas, although of course this ability declines once they're off the distribution, just like in humans. The model is creative and its output is transformative, however it only makes sense if you define creativity in a pure manner, based on novelty for itself.

However most people use entirely different definitions of creativity, something like "surprise me in a way that still makes sense to me". This includes "me", a side observer, and depends on what the side observer considers novel. Hence the main reason for the lack of "creativity" in big models is not some hand-waved "regression to the mean" or "interpolation", but the fact that they're still insufficiently intelligent compared to a human. Current models simply don't have enough fidelity to understand humans and think as deep, that's why humans think their output isn't sufficiently novel for them! Think about this, insanely smart aliens would probably dismiss humans as non-creative too.

The contributing factor is also the lack of semantic diversity in current models. Also known as mode collapse, but the name is a bit of misnomer and describes a technicality, not the resulting phenomenon. This indirectly affects creativity as it's usually understood, because the models generate and repeat the same thing in response to the same thing, which is the opposite of novel in layman's definition. That's part of where the slop comes from. Mode collapse has many causes, e.g. post-training with current algorithms. It's likely fixable but AI shops show little to no interest in studying and fixing it.