Funny thing is, LLMs can create novel ideas, they're just crap. Turn the temperature setting up or expand top_p and it'll come up with increasingly wacky responses, some of which might even be good.

Creativity often requires reasoning in unusual ways, and evaluating those ideas requires learning. The first part we can probably get LLMs to do; the latter part we can't (RL is a separate process and not really scalable).

Even without any of that, you can prompt your way into new things. I'm building a camper out of wood, and I've gotten older LLM models to make novel camper designs just by asking it questions and choosing things. You can make other AI models make novel music by prompting it to combine different aspects of music into a new song. Human creativity works that way too. Think of all the failed attempts at new things that humans come up with before a good one actually sticks.