Good points - mechanically just following algorithms isn't thinking, and neither is "predicting the next tokens".

But would a combination of the 2 then be close to what we define as thinking though?

> only means they have specialized, deeper contextualized information

no, LLMs can have that contextualized information. understanding in a reasoning sense means classifying the thing and developing a deterministic algorithm to process it. If you don't have a deterministic algorithm to process it, it isn't understanding. LLMs learn to approximate, we do that too, but then we develop algorithms to process input and generate output using a predefined logical process.

A sorting algorithm is a good example, when you compare that with an LLM sorting a list. they both may have correct outcome, but the sorting algorithm "understood" the logic and will follow that specific logic and have consistent performance.

> because we have no idea how it works

Flagrantly, ridiculously untrue. We don't know the precise nuts and bolts regarding the emergence of consciousness and the ability to reason, that's fair, but different structures of the brain have been directly linked to different functions and have been observed in operation on patients being stimulated in various ways with machinery attached to them reading levels of neuro-activity in the brain, and in specific regions. We know which parts handle our visual acuity and sense of hearing, and even cooler, we can watch those same regions light up when we use our "minds eye" to imagine things or engage in self-talk, completely silent speech that nevertheless engages our verbal center, which is also engaged by the act of handwriting and typing.

In short: no, we don't have the WHOLE answer. But to say that we have no idea is categorically ridiculous.

As to the notion of LLMs doing similarly: no. They are trained on millions of texts of various sources of humans doing thinking aloud, and that is what you're seeing: a probabilistic read of millions if not billions of documents, written by humans, selected by the machine to "minimize error." And crucially, it can't minimize it 100%. Whatever philosophical points you'd like to raise about intelligence or thinking, I don't think we would ever be willing to call someone intelligent if they just made something up in response to your query, because they think you really want it to be real, even when it isn't. Which points to the overall charade: it wants to LOOK intelligent, while not BEING intelligent, because that's what the engineers who built it wanted it to do.

Accepting as true "We don't know how the brain works in a precise way" does not mean that obviously untrue statements about the human brain cannot still be made. Your brain specifically, however, is an electric rat that pulls on levers of flesh while yearning for a taste of God's holiest cheddar. You might reply, "no! that cannot be!", but my statement isn't untrue, so it goes.

>>>There are no true and untrue claims about how the brain works, because we have no idea how it works.

Which is why if you pick up a neuroscience textbook it's 400 pages of blank white pages, correct?

There are different levels of understanding.

I don't need to know how a TV works to know there aren't little men and women acting out the TV shows when I put them on.

I don't need to know how the brain works in detail to know claims that humans are doing the same things as LLMs to be similarly silly.

Our output is quite literally the sum of our hardware (genetics) and input (immediate environment and history). For anyone who agrees that free will is nonsense, the debate is already over, we’re nothing more than output generating biological machines.