I always enjoy discussions that intersect between psychology and engineering.

But I feel this person falls short immediately, because they don't study neuroscience and psychology. That is the big gap in most of these discussion. People don't discuss things close to the origin.

We have to account for first principals in how intelligence works, starting from the origin of ideas and how humans process their ideas in novel ways that create amazing tech like LLM! :D

How Intelligence works

In Neuroscience, if you try to identify the origin of where and how thoughts are formed and how consciousness works. It is completely unknown. This brings up the argument, do humans have free will if we are driven by these thoughts of unknown origin? That's a topic for another thread.

Going back to intelligence. If you study psychology and what forms intelligence, there are many human needs that drive intelligence, namely intellectual curiosity (need to know), deprivation sensitivity (need to understand), aesthetic sensitivity, absorption, flow, openness to experience.

When you look at how a creative human with high intelligence uses their brain, there are 3 networks involved. Default mode network (imagination network), executive attention network and salience network.

The executive attention network controls the brains computational power. It has a working memory that can complete tasks using goal directed focus.

A person with high intelligence can alternate between their imagination and their working memory and pull novel ideas from their imagination and insert them into their working memory - frequently experimenting by testing reality. The salience network filters unnecessary content while we are using our working memory and imagination.

How LLMs work

Neural networks are quite promising in their ability to create a latent manifold within large datasets that interpolates between samples. This is the basis for generalization, where we can compress a large dataset in a lower dimensional space to a more meaningful representation that makes predictions.

The advent of attention on top of neural networks, to identify important parts of text sequences, is the huge innovation powering llms today. The innovation that emulates the executive attention network.

However, that alone is a long distance from the capabilities of human intelligence.

With current AI systems, there is the origin, which is known vocabulary with learned weights coming from neural networks, with reinforcement learning applied to enhance the responses.

Inference comes from an autoregressive sequence model that processes one token at a time. This comes with a compounding error rate with longer responses and hallucinations from counterfactuals.

Correct response must be in the training distribution.

As Andy Clark said, AI will never gain human intelligence, they have no motivation to interface with the world and conduct experiments and learn things on their own.

I think there are too many unknown and subjective components of human intelligence and motivation that cannot be replicated with the current systems.