That would be an example of, “overfitting.” It is something the AI labs are concerned about. They tweak parameters to try and avoid it.

It’s a hard balancing act to do. Give in too much randomness and you get non-sensical outputs that are difficult to align. Fit too closely to the training data and the model regurgitates the training data.

And oh, what’s that copyrighted material we never made any agreement to use doing in there?