So I do think there are two distinct types of activities involved in knowledge work:

1. Taking established techniques or concepts and appropriately applying them to novel situations.

2. Inventing or synthesizing new, never-before-seen techniques or concepts

The vast majority of the time, humans do #1. LLMs certainly do this in some contexts as well, as demonstrated by my example above. This to me counts as "understanding" and "thinking". Some people define "understanding" such that it's something only humans can do; to which I respond, I don't care what you call it, it's useful.

Can LLMs do #2? I don't know. They've got such extensive experience that how would you know if they'd invented a technique vs had seen it somewhere?

But I'd venture to argue that most humans never or rarely do #2.

> But I'd venture to argue that most humans never or rarely do #2.

That seems fair, although the distinction between synthesizing something new and combining existing techniques is a bit blurry.

What's missing from LLMs though is really part of 1). If techniques A, B, C & D are all the tools you need to solve a novel problem, then a human has the capability of learning WHEN to use each of these tools, and in what order/combination, to solve that problem - a process of trial and error, generalization and exception, etc. It's not just the techniques (bag of tools) you need, but also the rules (acquired knowledge) of how they can be used to solve different problems.

LLMs aren't able to learn at runtime from their own experience, so the only way they can learn these rules of when to apply given tools (aka reasoning steps) - is by RL training on how they have been successfully used to solve a range of problems in the training data. So, the LLM may have learnt that in specific context it should first apply tool A (generate that reasoning step), etc, etc, but that doesn't help it to solve a novel problem where the same solution step selection process doesn't apply, even if the tools A-D are all it needs (if only it could learn how to apply them to this novel problem).