This is a common rebuttal. But it falls flat, in my opinion, because LLMs appear to be a compression of language-based knowledge, and that compression is lossy. The result is that "knowledge" retrieved from the language-based storage model is biased towards the "average" idea.

As a concrete example, when you ask Claude Sonnet 5.5 to generate a small-to-medium-sized one-shot Python project, it will always fall into the same code patterns. Same structures of functions. Same file layout. Same package layout. They're not even good patterns, because if you throw a few common linters at it (e.g. flake8, isort, mypy) there are a ton of linting errors to clean up.

The only way that it escapes this mediocrity is through context. If you give it enough guardrails or sample data (e.g. Python code to emulate), it will start to go off in a "novel" direction that is defined by that context.

Now, what's the most likely way that "novelty" will be generated in an automated system like this? Randomly pulling in context and going for a random walk through the compressed language-based knowledge storage. It remains to be seen whether that's an effective way of producing more than just parlor tricks for investors. Human+LLM is overwhelmingly more novel and effective than only LLM.

If you want to argue, as I think you do, that there are no new ideas other than new combinations of old ideas, then you have to explain where the old ideas came from. It can't just be combinations all the way down. At some point, some initial set of ideas must have been created where none existed before otherwise we would have nothing to combine. After all the combinatoric closue of ∅ is ∅.

Ultimately any claim that there exist only combinations of old ideas must tackle this chicken-and-egg problem convincingly. For example- explain art. There was no art until it appeared fully formed and out of nowhere in early anatomically modern humans' dwellings (Neanderthals also had art but it was simpler and it is subject to the same chicken-and-eggness). So where did art come from? What pre-existing ideas were combined to bring it into existence from non-existence?

Btw, since I've had this conversation before- if the answer is that we looked around us in the world and saw things; yes, but we didn't see art. Where did art come from? Where did the idea come from of abstracting shapes and colours so that they resemble the shapes and colours of existing things, without actually being those things, or being composed of pieces of those things? How about numbers? How about writing? How about all our technology and science? Where did those things come from? What did we combine to create them?

Obviously I think that the argument that there are only new combinations is wrong: we are a creative species. We can create new things. Somehow. It would be very interesting to understand how. We have not yet created machines that can match us in that and we can't understand the "how" by studying the machines we have now. By denying the mystery of this our ability, by denying that it is a mystery, by dismissing it and accepting any facile explanation that presents itself in our current era -it's god, it's nature, we're like LLMs- we are making ourselves a great epistemic disservice, denying ourselves the chance to find out what it all means, to understand what kind of creature we truly are.

>> A calculator does math faster but we don't call it sentient or feel inferior or dismiss the results.

Actually, we do. We do dismiss calculator results. We are all generally aware of a calculator's abilities. We know it has limits. So when a calculator outputs 9999e999999 we know to distrust that result. It was something out of the calculators concept of understanding. So too a calculator's version of Pi. "3.14159265359" is not Pi and every user knows it is an approximation not be used for anything too important or precise. The same approach should apply to LLMs... but their limitations remain nebulous and ever-changing.

I agree with your opinion. The definition of novelty largely depends on subjectivity. Results produced by combinations of LLMs can also be novel. Combining multiple insights, imitating them, and then organizing them into one can also be considered sufficiently novel. LLMs have a kind of bias due to RLHF, and that may make them uninteresting to humans, but when a "human" uses them, they can enable the human to do more, and that greater amount of work can create new novelty.

The long name for this is Man Gelatin Ambrosia.

Known as the Gell-Mann Amnesia Effect.

The short name for that is Gell-Mann amnesia.

> build a car out of seatbelts

Such a good turn of phrase.

I have noticed that several times with a newspaper I had at the time. It seemed to be a generally good and reliable newspaper, and one of the biggest in the country, but every time an article was about something that I actually knew about, it was wrong, which made me doubt the validity of everything else.

I now have a newspaper that doesn't give me that feeling.

This is not novel. Computers have been generating mathematical information that would be resource-prohibitive to produce by hand since the dawn of computing.

The ingenious aspect seems to be that we've found simpler ways to apply computing power to a broader range of problems, which it's important to note is as much a testament to the developers of proof assistants like Lean as it is to LLM developers, to say nothing of the countless mathematicians over many centuries that contributed training data.

Whether this will lead to many revolutions, or a handful of impressive results followed by a long dark spell remains to be seen.

I, for one, wouldn't place large bets on either.

if you search papers with it, and from there search related papers and have it do literature reviews, it comes up with very useful white areas uncovered, and from there problem statements and experimental designs to uncover the white spots.

it is nothing but combining known knowledge that begets new.

Unless LLMs drastically shift incredibly quickly, said declaration is likely to hold, because of how LLMs work. I mean, they've worked the same way at their core for several years now.

This stokes my suspicion that effective LLM input for software development will ultimately resemble more of a higher level programming language than the natural language in common use today, and that the design of such languages and related tooling will play at least as large a role as the models themselves.

Most human jobs don't require innovation.

On the other hand, there's no evidence that work will dry up merely because it becomes more efficient, or even if it does, that this cannot lead to a better world. The trick is finding a way to prevent rapid changes in the nature of work from destroying too many lives, and that's a very human problem machines alone can't solve.

> I'd argue it is the other way around - LLMs are great at pure creativity

Interesting because pure creativity is the most prominent thing they can't do by design.

They can be quite good at taking knowledge from one domain and applying it to another. This can give new perspectives on things, but it isn't explicitly new information.

I too am entertained by LLMs.

Morbid fascination.

What?

They literally generate stuff that's averages of other stuff.

It's not interesting in the least

I guess you chose a fitting username

They're terrible at jokes. I bet any good joke they've given you is simply new TO YOU.

I have a PhD and Opus 5.5 came up with a novel solution to solve a problem in my area of research that I, and other experts in the field, had been unable to. This was from very general prompting it to find improvements over published methods.

>If you're an expert on the topic, framework, or library, even the best LLMs are still underwhelming.

How many people on this planet are experts on every topic?

>An LLM is like a low IQ person with vast knowledge.

Are you aware that frontier models have scored in the 135-195 IQ range when they have been tested? How smart are you?

Does this gel with top mathematicans basically begging OpenAI to stop solving their open problems?

That's not really true any more, since given tool use and iterative 'chain of thought' reasoning, it's more like an answer refined by conversation with the asker or the second best answerer, from the best answerer, is the best that you can expect.

E.g. consider the best answer describes how to achieve something with some CLI, but the CLI's changed and the command is now broken. It can discover that, combine with the 'best answer' about what changed in the CLI, and fix it to work for the original problem even if that's completely novel and there were no extant answers about solving the problem with the new CLI form, or that it had broken that use case.

It doesn't change fundamentally how they work of course, but 'thinking', creating that feedback loop, made them significantly more useful IMO.

> The very best LLM can answer is the answer you can expect from an online forum like Reddit or a book/paper published more than a year ago.

Hard disagree.

Claude is able to come up with solutions in very niche areas where there is little discussion on the web, because it takes steps to digest the source of (for instance) frameworks and libraries, and can then use/advise on lots of esoteric options, edge-cases and all sorts of stuff that doesn't get discussed on the open web.

You’re wrong on literally every point.

Nobody solves the Quasi-Riemann Hypothesis by looking up online forums.

>The very best LLM can answer is the answer you can expect from an online forum like Reddit or a book/paper published more than a year ago.

The LLMs are vastly superior for any subject matter with even the slightest hint of danger or otherwise high stakes because while both LLMs and Reddit will whine at you and talk down to you the LLM will actually knock it off when told.

Like I'm asking you about sheetmetal thickness vs joist spacing for re-skinning a goddamn carport with roofing steel, read the room, this is not the time to whine about permits and snow loads and whatnot because the project is happening with or without your advice.

And it's not just mundane stuff. Some guy who's retrofitting new chinesium controls onto 50yo industrial equipment that had a prior generation of controls cobbled on 20yr ago doesn't need to get talked down at about stainless on stainless corrosion when he's asking a question about a solenoid or something. If the LLM talks down to him he can brush it aside. If he tries that on reddit they'll dig in their heels and fling poo.

Cutting out the "negative value" middle man of having to painstakingly make the case about the validity of your big picture goal before getting answers to simple questions is of substantial value. Stuff that people would've painstakingly researched themselves rather than ask Reddit or HN or Stack Overflow because dealing with those people is so onerous they now just punch into their LLM of choice and get the best of both worlds.