One argument for thinking being different from computing is that thought is fundamentally embodied, conscious and metaphorical. Computing would be an abstracted activity from thinking that we've automated with machines.

[dead]

except for the implications of one word over another are world-changing

They can be made boring by reducing them to an arbitrary choice of definition of the word "thinking", but the question is really about weather inference is in principle as powerful as human thinking, and so would deserve to be applied the same label. Which is not at all a boring question. It's equivalent to asking weather current architectures are enough to reach AGI.

Reading the quote in context seems to indicate Dijkstra meant something else. His article is a complaint about overselling computers as doing or augmenting the thinking for humans. It's funny how the quote was lifted out of an article and became famous on it's own.

https://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD867...

Then you're missing the point of my rebuttal. You say submarines don't swim [like fish] despite both moving through water, the only distinction is mechanism. Can AI recursively create new capabilities like thinking does, or just execute tasks like submarines do? That's the question.

It's also nonsense. (Swimming and thinking are both human capabilities, not solutions to problems.)

But of course here we are back in the endless semantic debate about what "thinking" is, exactly to the GP's (and Edsger Dijkstra's) point.

Wrong. The same exact model can do both, depending on the circumstances.

> Humans don't require input to, say, decide to go for a walk.

Impossible to falsify since humans are continuously receiving inputs from both external and internal sensors.

> What's missing in the LLM is volition.

What's missing is embodiment, or, at least, a continuous loop feeding a wide variety of inputs about the state of world. Given that, and info about of set of tools by which it can act in the world, I have no doubt that current LLMs would exhibit some kind (possibly not desirable or coherent, from a human POV, at least without a whole lot of prompt engineering) of volitional-seeming action.

Our entire extistence and experience is nothing _but_ input.

Temperature changes, visual stimulus, auditory stimulus, body cues, random thoughts firing, etc.. Those are all going on all the time.

LLMs can absolutely generate output without input but we don’t have zero input. We don’t exist in a floating void with no light or sound or touch or heat or feelings from our own body.

But again this doesn’t see to be the same thing as thinking. If I could only reply to you when you send me a message but could reason through any problem we discuss just like “able to want a walk” me could, would that mean I no longer could think? I think these are different issues.

On that though, these see trivially solvable with loops and a bit of memory to write to and read from - would that really make the difference for you? A box setup to run continuously like this would be thinking?

It's as if a LLM is only one part of a brain, not the whole thing.

So of course it doesn't do everything a human does, but it still can do some aspects of mental processes.

Whether "thinking" means "everything a human brain does" or whether "thinking" means a specific cognitive process that we humans do, is a matter of definition.

I'd argue that defining "thinking" independently of "volition" is a useful definition because it allows us to break down things in parts and understand them

> Humans don't require input to, say, decide to go for a walk.

Very much a subject of contention.

How do you even know you're awake, without any input?

I would not say it is missing but thankfully absent.

As a researcher in these fields: this reasoning is tired, overblown, and just wrong. We have a lot of understanding of how the brain works overall. You don't. Go read the active inference book by Friston et. al. for some of the epistemological and behavioral mechanics (Yes, this applies to llms as well, they easily satisfy the requirements to be considered the mathematical object described as a markov blanket).

And, yes, if you could somehow freeze a human's current physical configuration at some time, you would absolutely, in principle, given what we know about the universe, be able to concretely map input to into actions. You cannot separate a human's representative configuration from their environment in this way, so, behavior appears much more non-deterministic.

Another paper by Friston et al (Path Integrals, particular kinds, and strange things) describes systems much like modern modeling and absolutely falls under the same action minimization requirements for the math to work given the kinds of data acquisition, loss functions, and training/post-training we're doing as a research society with these models.

I also recommend https://arxiv.org/abs/2112.04035, but, in short, transformer models have functions and emergent structures provably similar both empirically and mathematically to how we abstract and consider things. Along with https://arxiv.org/pdf/1912.10077, these 4 sources, alone, together strongly rebuke any idea that they are somehow not capable of learning to act like and think like us, though there's many more.

> Yet we can ignore our biology, or act in ways that are the opposite of what our biology tells us.

I have Coeliac disease, in that specific case I'd really love to be able to ignore what "my biology" tells my body to do. I'd go eat all the things I know wouldn't be good for me to eat.

Yet I fear "my biology" has the upper hand :/

Good luck ignoring your biology’s impulse to breathe

I know what you mean but was just thinking people vary a lot in their requirements as to what they will accept as thinking. People show a kid a photo and say what's that and they say I think it's a dog and that's taken as evidence of thinking. With AI people want it to win a Nobel prize or something.

I'll concede to all your points here, but I was nevertheless extremely impressed by this result.

You're right of course that this was not without human guidance but to me even successfully using LLMs just for the mutation step was in and of itself surprising enough that it revised my own certainty that llms absolutely cannot think.

I see this more like a step in the direction of what you're looking for, not as a counter example.

Yes, it's a very technical and circumscribed result, not requiring a deep insight into the nature of various mathematical models.

It's unclear whether intelligence, consciousness and capacity for suffering are linked in any way - other than by that all three seem to coincide in humans. And the nature of consciousness does not yield itself to instrumentation.

It's also worth noting that there's a lot of pressure to deny that "intelligence", "consciousness" or "capacity for suffering" exist in LLMs. "AI effect" alone demands that all three things should remain human-exclusive, so that humans may remain special. Then there's also an awful lot of money that's riding on building and deploying AIs - and money is a well known source of cognitive bias. That money says: AIs are intelligent but certainly can't suffer in any way that would interfere with the business.

Generally, the AI industry isn't at all intrigued by the concept of "consciousness" (it's not measurable), and pays very limited attention to the idea of LLMs being potentially capable of suffering.

The only major company that seems to have this consideration is Anthropic - their current plan for "harm reduction", in case LLMs end up being capable of suffering, is to give an LLM an "opt out" - a special output that interrupts the processing. So that if an LLM hates doing a given task, it can decide to not do it.

Emotions tend to manifest as physical sensations, and if you don't think that's true it's likely you haven't been paying attention. See also https://www.theatlantic.com/health/archive/2013/12/mapping-h...

Do you think animals don't have feelings?

The black box I'm referring to is us.

You can't have it both ways. If your test for whether something is intelligent/thinking or not isn't applicable to any known form of intelligence, then what you are testing for is not intelligence/thinking.

You wouldn't say this about a message encrypted with AES though, since there's not just a "human patience" limit but also a (we are pretty sure) unbearable computational cost.

We don't know, but it's completely plausible that we might find that the cost of analyzing LLMs in their current form, to the point of removing all doubt about how/what they are thinking, is also unbearably high.

We also might find that it's possible for us (or for an LLM training process itself) to encrypt LLM weights in such a way that the only way to know anything about what it knows is to ask it.

Just because it runs on a computer doesn’t mean it’s “software” in the common meaning of the word

> But language is the input and the vector space within which their knowledge is encoded and stored. The don't have a concept of a duck beyond what others have described the duck as.

I guess if we limit ourselves to "one-modal LLMs" yes, but nowadays we have multimodal ones, who could think of a duck in the way of language, visuals or even audio.

What if we learned that brains reduce to the same basic mechanics?

What is an idea?