What then is your LLM "thinking" about between answers? The answer is nothing. Your definition of thinking does not match the one humans normally use.

>That how we know another person is thinking too. By their output. We don't put a debugger into their brain.

We know thoughts exist in their brain between the ones they choose to verbalize. Avoiding the distraction of solipsism.

For the LLM the "thinking" phase is just a preamble output for creating the answer. It just gets appended to the context window. Remove the context windows from your models and you will see how much of a mind they truly have. None.

>What then is your LLM "thinking" about between answers? The answer is nothing.

Between answer it's thinking something else, somebody else asked :) You think that hardware sits idle?

That aside, what is a human thinking while unconscious? Does having been unconscious (e.g. for an operation, or fainting or whatever) means somebody doesn't think in general?

>We know thoughts exist in their brain between the ones the choose to verbalize

And we also know that if we run an LLM in a loop, didn't give it a cutoff for stopping their output, and didn't force it to print everything in the end, thoughts would exist in their "brain" too between the ones they chose to verbalize.

In fact, that's exactly how some LLMs in "thinking mode" appear.