As you say, putting a random sequence of plausible-sounding words together is a trivial task - even a non-LLM algorithm can do it. No, I don't think humans are unique - they also absorb the ideas available to them and re-elaborate on them to produce novel ones.

An LLM might come up with a sequence of words that express a novel idea/thinking pattern, but so can I on the spot right now. The question, though, is: is that idea any good? In the case of humans, maybe the first idea is good, the second ok-ish, but inevitably they get duller.

Most people come up with very few novel (good) ideas in their lifetime. The question, then, is whether a model can, unlike humans, *repeatedly* generate new ideas, continuously.

My guess/assumption is that NO, a model might come up with SOME ideas that are somewhat original [1], mixing the existing sources available to it (similarly to how a human would do it, just with more sources and more efficiently), but it cannot continuously generate novel ideas.

If that's true, I think there will be a space for humans to be thinkers, even if models are super smart, as a different unique perspective and a different reconfiguration of previous ideas will be more unique than raw intelligence.

Please note: I wasn't refusing the notion that LLMs don't produce anything original, just noting some fallacy in your example about word generation. The jury is still out on whether AI models will own thinking in the future - or will simply be partners to humans - and I don't pretend to have the answer.

[1] More should be said about what makes an idea original (even if derived from others), as opposed to a transposition of an existing concept. I admit I'm not a mathematician or anywhere near an expert on the topic, but from what I've read, the latest AI math achievements seem to belong to the latter case.

As Stalin said, quantity has a quality all it's own.

11000 agents explored one of the recent math problems solved by AI (can't remember exactly which one). About 3 of them had crucial insights which moved the problem forward. A 4th one also had one of the key ideas, but it's session got messed up and the idea was lost, and only noticed in retrospect.

Maybe 11000 top mathematicians working on it would have found the same ideas. But certainly not in days.

Even if LLMs generate novel ideas at 0.01 rate of humans, you can deploy vast amounts of them on even trivial problems, that humans would not bother with.

For biological reasons, human intelligence is kind of fixed and only very slowly advances by better compression and synthesis of the school curriculum.

There is no clear present bound on machine intelligence. One might argue that machines won't be able to get smarter than the material they were trained on (human output). I kind of doubt that. And certainly they can be much faster.