I think my point is that when the doctor diagnoses you, she often doesn’t do so immediately. She is spending time thinking it through, and as part of that process is retrieving various pieces of relevant information from her memory (both long term and short term).

I think this may be closer to an agentic, iterative search (ala claude code) than direct inference using continuously updated weights. If it was the latter, there would be no process of thinking it through or trying to recall relevant details, past cases, papers she read years ago, and so on; the diagnosis would just pop out instantaneously.

Yes, but I think a key part of learning is experimentation and the feedback loop of being wrong.

An agent, or doctor, may be reasoning over the problem they are presented with, combining past learning with additional sources of memorized or problem-specific data, but in that moment it's their personal expertise/learning that will determine how successful they are with this reasoning process and ability to apply the reference material to the matter at hand (cf the plumber, who with all the time in the world just doesn't have the learning to make good use of the reference books).

I think there is also a subtle problem, not often discussed, that to act successfully, the underlying learning in choosing how to act has to have come from personal experience. It's basically the difference between being book smart and having personal experience, but in the case of an LLM also applies to experience-based reasoning it may have been trained on. The problem is that when the LLM acts, what is in it's head (context/weights) isn't the same as what was in the head of the expert whose reasoning it may be trying to apply, so it may be trying to apply reasoning outside of the context that made it valid.

How you go from being book smart, and having heard other people's advice and reasoning, to being an expert yourself is by personal practice and learning - learning how to act based on what is in your own head.