> "If what LLMs do today isn't actual thinking, what is something that only an actually thinking entity can do that LLMs can't?"

Independent frontier maths research, i.e. coming up with and proving (preferably numerous) significant new theorems without human guidance.

I say that not because I think the task is special among human behaviours. I think the mental faculties that mathematicians use to do such research are qualitatively the same ones all humans use in a wide range of behaviours that AI struggles to emulate.

I say it because it's both achievable (in principle, if LLMs can indeed think like humans) and verifiable. Achievable because it can be viewed as a pure text generation task and verifiable because we have well-established, robust ways of establishing the veracity, novelty and significance of mathematical claims.

It needs to be frontier research maths because that requires genuinely novel insights. I don't consider tasks like IMO questions a substitute as they involve extremely well trodden areas of maths so the possibility of an answer being reachable without new insight (by interpolating/recombining from vast training data) can't be excluded.

If this happens I will change my view on whether LLMs think like humans. Currently I don't think they do.

▲

pegasus 3 days ago | parent | next [-]

This, so much. Many mathematicians and physicists believe in intuition as a function separate from intelect. One is more akin to a form of (inner) perception, whereas the other is generative - extrapolation based on pattern matching and statistical thinking. That second function we have a handle on and getting better at it every year, but we don't even know how to define intuition properly. A fascinating book that discusses this phenomena is Nature Loves to Hide: Quantum Physics and Reality, a Western Perspective [1]

This quote from Grothendieck [2] (considered by many the greatest mathematician of the 20th century) points to a similar distinction: The mathematician who seeks to understand a difficult problem is like someone faced with a hard nut. There are two ways to go about it. The one way is to use a hammer — to smash the nut open by brute force. The other way is to soak it gently, patiently, for a long time, until it softens and opens of itself.

[1] https://www.amazon.com/Nature-Loves-Hide-Quantum-Perspective...

[2] https://en.wikipedia.org/wiki/Alexander_Grothendieck

▲

tim333 3 days ago | parent | prev | next [-]

That's quite a high bar for thinking like humans which rules out 99.99% of humans.

▲

omnicognate 3 days ago | parent [-]

I have never claimed that only people/machines that can do frontier maths research can be intelligent. (Though someone always responds as if I did.)

I said that a machine doing frontier maths research would be sufficient evidence to convince me that it is intelligent. My prior is very strongly that LLM's do not think like humans so I require compelling evidence to conclude that they do. I defined one such possible piece of evidence, and didn't exclude the possibility of others.

If I were to encounter such evidence and be persuaded, I would have to also consider it likely that LLMs employ their intelligence when solving IMO questions and generating code. However, those tasks alone are not sufficient to persuade me of their intelligence because I think there are ways of performing those tasks without human-like insight (by interpolating/recombining from vast training data).

As I said elsewhere in this thread:

> The special thing about novel mathematical thinking is that it is verifiable, requires genuine insight and is a text generation task, not that you have to be able to do it to be considered intelligent.

▲

tim333 2 days ago | parent [-]

I know what you mean but was just thinking people vary a lot in their requirements as to what they will accept as thinking. People show a kid a photo and say what's that and they say I think it's a dog and that's taken as evidence of thinking. With AI people want it to win a Nobel prize or something.

	▲	omnicognate 2 days ago \| parent [-]
		It's about priors again. I don't need evidence that humans think like humans. My prior on that is absolute certainty that they do, by definition. If, on the other hand, you wanted to persuade me that the kid was using an image classifier trained by backpropagation and gradient descent to recognise the dog I'd require strong evidence.

▲

OrderlyTiamat 3 days ago | parent | prev [-]

Google's AlphaEvolve independently discovered a novel matrix multiplication algorithm which beats SOTA on at least one axis: https://www.youtube.com/watch?v=sGCmu7YKgPA

▲

omnicognate 3 days ago | parent [-]

That was an impressive result, but AIUI not an example of "coming up with and proving (preferably numerous) significant new theorems without human guidance".

For one thing, the output was an algorithm, not a theorem (except in the Curry-Howard sense). More importantly though, AlphaEvolve has to be given an objective function to evaluate the algorithms it generates, so it can't be considered to be working "without human guidance". It only uses LLMs for the mutation step, generating new candidate algorithms. Its outer loop is a an optimisation process capable only of evaluating candidates according to the objective function. It's not going to spontaneously decide to tackle the Langlands program.

Correct me if I'm wrong about any of the above. I'm not an expert on it, but that's my understanding of what was done.

	▲	OrderlyTiamat 3 days ago \| parent \| next [-]
		I'll concede to all your points here, but I was nevertheless extremely impressed by this result. You're right of course that this was not without human guidance but to me even successfully using LLMs just for the mutation step was in and of itself surprising enough that it revised my own certainty that llms absolutely cannot think. I see this more like a step in the direction of what you're looking for, not as a counter example.
	▲	pegasus 3 days ago \| parent \| prev [-]
		Yes, it's a very technical and circumscribed result, not requiring a deep insight into the nature of various mathematical models.