For me, the way humans finish each other's sentences and often think of quotes from the same movies at the same time in conversation (when there is no clear reason for that quote to be a part of the conversation), indicates that there is a probabilistic element to human thinking.

Is it entirely probabilistic? I don't think so. But, it does seem that a chunk of our speech generation and processing is similar to LLMs. (e.g. given the words I've heard so far, my brain is guessing words x y z should come next.)

I feel like the conscious, executive mind humans have exercises some active control over our underlying probabilistic element. And LLMs lack the conscious executive.

e.g. They have our probabilistic capabilities, without some additional governing layer that humans have.

I think the better way to look at it is that probabilistic models seem to be an accurate model for human thought. We don't really know how humans think, but we know that they probably aren't violating information theoretic principles, and we observe similar phenomena when we compare humans with LLMs.