Firstly, it doesn't really matter if they can produce novel designs or not. 99% of what is being done is not novel. It is manipulating data in ways computers have been manipulating data for decades. The design of what is implemented is also going to be derivative of what already exists in the world too. Being too novel makes for a bad product since users will not easily understand how to use it and adapt their existing knowledge of how other things work.

Secondly, they are able to produce intelligence that wasn't represented in their training input. As a simple example take a look at chess AI. The top chess engines have more intelligence over the game of chess than the top humans. They have surpassed humans understanding of chess. Similar with LLMs. They train on synthetic data that other LLMs have made and are able to find ways to get better and better on their own. Humans learn off the knowledge of other humans and it compounds. The same thing applies to AI. It is able to generated information and try things and then later reference what it tried when doing something else.

Chess AI isn't trained in the same way. Things like alpha zero partly worked by playing themselves recursively, meaning they actually did generate novel data in the process.

That was partly possible because chess is a constrained domain: rigid rules and board states.

But LLM land is not like that. LLM land was trained on pre-existing text written by humans. They do discover patterns within said data but the point stands that the data and patterns within are not actually novel.