> However, LLMs will not be able to represent ideas that it has not encountered before.

Sure they do. We call them hallucinations and complain that they're not true, however.

Hmmm. Didn't think about that.

In people there is a difference between unconscious hallucinations vs. intentional creativity. However, there might be situations where they're not distinguishable. In LLMs, it's hard to talk about intentionality.

I love where you took this.

Hallucinations are not novel ideas. They are novel combinations of tokens constrained by learned probability distributions.

I have mentioned Hume before, and will do so again. You can combine "golden" and "mountain" without seeing a golden mountain, but you cannot conjure "golden" without having encountered something that gave you the concept.

LLMs may generate strings they have not seen, but those strings are still composed entirely from training-derived representations. The model can output "quantum telepathic blockchain" but each token's semantic content comes from training data. It is recombination, not creation. The model has not built representations of concepts it never encountered in training; it is just sampling poorly constrained combinations.

Can you distinguish between a false hallucination and a genuinely novel conceptual representation?

Or, 10000000s times a day while coding all over the world and it hallucinating something it never saw before which turned out to be the thing needed.