I don’t think that is a good example. No one is debating whether LLMs can generate completely new sequences of tokens that have never appeared in any training dataset. We are interested not only in novel output, we are also interested in that output being correct, useful, insightful, etc. Copying a sequence from the user’s prompt is not really a good demonstration of that, especially given how autoregression/attention basically gives you that for free.

Perhaps I should have quoted the parent:

> That means the group of characters it outputs must have been quite common in the past. It won't add a new group of characters it has never seen before on its own.

My only claim is that precisely this is incorrect.