Sometimes after a night’s sleep, we wake up with an insight on a topic or a solution to a problem we encountered the day before. Did we “think” in our sleep to come up with the insight or solution? For all we know, it’s an unconscious process. Would we call it “thinking”?

The term “thinking” is rather ill-defined, too bound to how we perceive our own wakeful thinking.

When conversing with LLMs, I never get the feeling that they have a solid grasp on the conversation. When you dig into topics, there is always a little too much vagueness, a slight but clear lack of coherence, continuity and awareness, a prevalence of cookie-cutter verbiage. It feels like a mind that isn’t fully “there” — and maybe not at all.

I would agree that LLMs reason (well, the reasoning models). But “thinking”? I don’t know. There is something missing.

▲ AnIrishDuck 3 days ago | parent | next [-]

> Sometimes after a night’s sleep, we wake up with an insight on a topic or a solution to a problem we encountered the day before.

The current crop of models do not "sleep" in any way. The associated limitations on long term task adaptation are obvious barriers to their general utility.

> When conversing with LLMs, I never get the feeling that they have a solid grasp on the conversation. When you dig into topics, there is always a little too much vagueness, a slight but clear lack of coherence, continuity and awareness, a prevalence of cookie-cutter verbiage. It feels like a mind that isn’t fully “there” — and maybe not at all.

One of the key functions of REM sleep seems to be the ability to generalize concepts and make connections between "distant" ideas in latent space [1].

I would argue that the current crop of LLMs are overfit on recall ability, particularly on their training corpus. The inherent trade-off is that they are underfit on "conceptual" intelligence. The ability to make connections between these ideas.

As a result, you often get "thinking shaped objects", to paraphrase Janelle Shane [2]. It does feel like the primordial ooze of intelligence, but it is clear we still have several transformer-shaped breakthroughs before actual (human comparable) intelligence.

1. https://en.wikipedia.org/wiki/Why_We_Sleep 2. https://www.aiweirdness.com/

▲ popalchemist 2 days ago | parent | prev | next [-]

There is simply put no ongoing process and no feedback loop. The model does not learn. The cognition ends when the inference cycle ends. It's not thinking, it just produces output that looks similar to the output of thinking. But the process by which it does that is wholly unreleated.

▲ AlphaAndOmega0 2 days ago | parent | next [-]

Humans with certain amnestic syndromes are incapable of learning. That doesn't make them unintelligent or incapable of thought.

▲ ako 2 days ago | parent | prev | next [-]

Most AI tooling is shipped with a feedback loop around the LLM. The quality of Claude Code for example lies in the feedback loop it provides on your code. Maybe the LLM itself isn't thinking, but the Agent which ships an LLM plus feedback loop definitely shows thinking qualities.

Just now in an debugging session with claude code:

  * let me read this file...
  * let me read this file...
  * I think there's a caching issue with the model after dropping the module. Let me check if there's a save or reload needed after DROP MODULE. First, let me verify something:
  * creates a bash/javascript script to verify its assumption
  * runs the script (after review and approval)
  * Aha! I found the problem! Look at the output...

How is this not thinking?

▲

popalchemist 2 days ago | parent [-]

without getting into theory of mind it's a bit difficult to elaborate, and I don't have the time or the will for that. But the short version is that thinking is interconnected with BEING as well as will, and the Agent has neither, in a philosophically formal sense. The agent is deterministically bound. So it is a fancy Rube Goldberg machine that outputs letters in a way that creates the impression of thought, but it is not thought, in the same way that some birds can mimic human speech without even the slightest hint as to the words' or sentences' meaning, underlying grammar, connotations, subtext, context, intended use, likely effect, etc. Is speech speech if the speaker has no concept whatsoever of said speech's content, and can not use it to actualize itself? I'd say no. It's mimicry, but not speech. So that means speech is something more than just its outward aspect - the words. It is the relation of something invisible, some inner experience known only to the speaker, VIA the words.

Whereas a gorilla who learns sign language to communicate and use that communication to achieve aims which have direct correlation with its sense of self - that's thought in the Cogito, Ergo Sum sense of the word.

Thought as commonly concieved by the layman is a sort of isolated phenomenon that is mechanical in nature and can be judged by its outward effects; whereas in the philosophical tradition defining thought is known to be one of the hard questions for its mysterious qualia of being interconnected with will and being as described above.

Guess I gave you the long answer. (though, really, it could be much longer than this.) The Turing Test touches on this distinction between the appearance of thought and actual thought.

The question goes all the way down to metaphysics; some (such as myself) would say that one must be able to define awareness (what some call consciousness - though I think that term is too loaded) before you can define thought. In fact that is at the heart of the western philosophical tradition; and the jury consensus remains elusive after all these thousands of years.

▲

TheOtherHobbes 2 days ago | parent | next [-]

The obvious counterargument is that a calculator doesn't experience one-ness, but it still does arithmetic better than most humans.

Most people would accept that being able to work out 686799 x 849367 is a form of thinking, albeit an extremely limited one.

First flight simulators, then chess computers, then go computers, then LLMs are the same principle extended to much higher levels of applicability and complexity.

Thinking in itself doesn't require mysterious qualia. It doesn't require self-awareness. It only requires a successful mapping between an input domain and an output domain. And it can be extended with meta-thinking where a process can make decisions and explore possible solutions in a bounded space - starting with if statements, ending (currently) with agentic feedback loops.

Sentience and self-awareness are completely different problems.

In fact it's likely with LLMs that we have off-loaded some of our cognitive techniques to external hardware. With writing, we off-loaded memory, with computing we off-loaded basic algorithmic operations, and now with LLMs we have off-loaded some basic elements of synthetic exploratory intelligence.

These machines are clearly useful, but so far the only reason they're useful is because they do the symbol crunching, we supply the meaning.

From that point of view, nothing has changed. A calculator doesn't know the meaning of addition, an LLM doesn't need to know the meaning of "You're perfectly right." As long as they juggle symbols in ways we can bring meaning to - the core definition of machine thinking - they're still "thinking machines."

It's possible - I suspect likely - they're only three steps away from mimicking sentience. What's needed is a long-term memory, dynamic training so the model is constantly updated and self-corrected in real time, and inputs from a wide range of physical sensors.

At some point fairly soon robotics and LLMs will converge, and then things will get interesting.

Whether or not they'll have human-like qualia will remain an unknowable problem. They'll behave and "reason" as if they do, and we'll have to decide how to handle that. (Although more likely they'll decide that for us.)

	▲	popalchemist 2 days ago \| parent \| next [-]
		Some of your points are lucid, some are not. For example, an LLM does not "work out" any kind of math equation using anything approaching reasoning; rather it returns a string that is "most likely" to be correct using probability based on its training. Depending on the training data and the question being asked, that output could be accurate or absurd. That's not of the same nature as reasoning your way to an answer.
	▲	ako 2 days ago \| parent \| prev [-]
		So if you don’t have a long term memory, you’re not capable of sentience? Like the movie memento, where the main character needs to write down everything to remind him later because he’s not able to remember anything. This is pretty much like llms using markdown documents to remember things.

▲

Workaccount2 2 days ago | parent | prev | next [-]

"To escape the paradox, we invoke what we call the “Homunculus Defense”: inside every human is a tiny non-stochastic homunculus that provides true understanding. This homunculus is definitionally not a stochastic parrot because:

1. It has subjective experience (unprovable but assumed)

2. It possesses free will (compatibilist definitions need not apply)

3. It has attended at least one philosophy seminar"[1]

[1]https://ai.vixra.org/pdf/2506.0065v1.pdf

▲

ako 2 days ago | parent | prev | next [-]

For practical every day uses, does it really matter if it is "real thinking" or just really good "artificial thinking" with the same results? The machine can use artificial thinking to reach desired goals and outcomes, so for me it's the kind of thinking i would want from a machine.

	▲	popalchemist 2 days ago \| parent [-]
		For practical purposes, abstractions are useful, so, no, it doesn't matter. But the post we are responding to is directly posing the philosophical question of whether the activity of LLM agents constitutes thought.

▲

Earw0rm 2 days ago | parent | prev [-]

It seems pretty clear to me though that being good at intellectual tasks / the sort of usefulness we ascribe to LLMs doesn't strongly correlate with awareness.

Even just within humans - many of the least intellectually capable humans seem to have a richer supply of the traits associated with awareness/being than some of the allegedly highest-functioning.

On average you're far more likely to get a sincere hug from someone with Down's syndrome than from a multi-millionaire.

But I'm more interested in this when it comes to the animal kingdom, because while ChatGPT is certainly more useful than my cat, I'm also pretty certain that it's a lot less aware. Meaningful awareness - feelings - seems to be an evolutionary adaptation possessed by k-strategy reproducing vertebrates. Having a small number of kids and being biologically wired to care for them has huge implications for your motivation as an animal, and it's reasonable to think that a lot of our higher emotions are built on hardware originally evolved for that purpose.

(Albeit the evolutionary origins of that are somewhat murky - to what extent mammals/birds reuse capabilities that were developed by a much earlier common ancestor, or whether it's entirely parallel evolution, isn't known afaik - but birds seem to exhibit a similar set of emotional states to mammals, that much is true).

▲ Festro 2 days ago | parent | prev [-]

You're moving the goalposts and contradicting yourself with language games.

Something doesn't need to learn to think. I think all the time without learning.

There's also an argument for machines already starting to crack learning with literal reinforcement training and feedback loops.

Your language game was when you said the 'cognition ends...', as cognition is just a synonym for thinking. "The thinking ends when the inference cycle ends. It's not thinking'" becomes a clear contradiction.

As for "the process by which it does that is wholly unrelated", buddy it's modelled on human neuron behaviour. That's how we've had this generative AI breakthrough. We've replicated human mental cognition as closely as we can with current technology and the output bears striking resemblance to our own generative capabilities (thoughts).

Happy to admit it's not identical, but it's damn well inside the definition of thinking, may also cover learning. It may be better to take a second look at human thinking and wonder if it's as cryptic and deep as we thought ten, twenty years ago.

▲ creer 3 days ago | parent | prev | next [-]

Interesting, you think the associations your brain comes up with during sleep are NOT thinking?

▲

madaxe_again 3 days ago | parent | next [-]

I’d say it’s more like annealing. Sort, filter, compress, integrate.

Updates your models for the next morning, which is why the answer is there when it wasn’t before.

▲

keiferski 3 days ago | parent | prev [-]

I assume most people agree with that framing. “Thinking” generally implies a level of consciousness or self-awareness.

“Let me think about this.” “I have to think on it.”

My brain regulates all sorts of processes unconsciously, like breathing, for example. I don’t treat those as “thinking,” so I don’t know why other unconscious brain activity would be either.

	▲	creer 2 days ago \| parent [-]
		I don't know that most people think about that at all. For me, I have learned not to dismiss the remuninating, reordering, reframing that my brain performs without me deliberately reasoning line by line about it. Limiting my brain to deliberate line by line reasoning would be very limiting and costly. "Thinking" to me is very much NOT just conscious reasoning. So much of what I think is not done consciously. Indeed "let me think about it" is often simply giving my brain time to "sit on it", for another expression - only after which will I have enough mind time on the various alternatives for a worthwhile conscious decision.

▲ madaxe_again 3 days ago | parent | prev | next [-]

Perhaps this is an artefact of instantiation - when you talk with an LLM, the responding instance is just that - it comes into being, inhales your entire chat history, and then continues like the last chap, finishes its response, and dies.

The continuity is currently an illusion.

▲ bithead 3 days ago | parent | prev | next [-]

Do LLMs ever ask for you to clarify something you said in a way a person who doesn't quite understand what you said will do?

	▲	Workaccount2 2 days ago \| parent \| next [-]
		Sometimes I think people leveraging criticisms of LLMs used ChatGPT 3 years ago and haven't touched one since, except for asking how many r's are in strawberry a year and a half ago.
	▲	savolai 3 days ago \| parent \| prev \| next [-]
		Yeah, as someone who has gained a lot of interaction skills by playing with the constructivist learning ennvironment called the enneagram, I can attest that it much resembles behaviour characteristic of certain enneatypes.
	▲	willmarch 3 days ago \| parent \| prev \| next [-]
		Yes, often
	▲	brabel 3 days ago \| parent \| prev [-]
		What now, two minutes using one and you are going to get that!

▲ lordnacho 2 days ago | parent | prev | next [-]

Much like speaking to a less experienced colleague, no?

They say things that contain the right ideas, but arrange it unconvincingly. Still useful to have though.

▲ petralithic 3 days ago | parent | prev [-]

> Would we call it “thinking”?

Yes I would.