> Any answer other than "that was the most likely next token given the context" is untrue.

"Because the matrix math resulted in the set of tokens that produced the output". "Because the machine code driving the hosting devices produced the output you saw". "Because the combination of silicon traces and charges on the chips at that exact moment resulted in the output". "Because my neurons fired in a particular order/combination".

I don't see how your statement is any more useful. If an LLM has access to reasoning traces it can realistically waddle down the CoT and figure out where it took a wrong turn.

Just like a human does with memories in context - does't mean that's the full story - your decision making is very subconscious and nonverbal - you might not be aware of it, but any reasoning you give to explain why you did something is bound to be an incomplete story, created by your brain to explain what happened based on what it knows - but there's hidden state it doesn't have access to. And yet we ask that question constantly.

well, do you want something useful or something true?

the word why is used to get something true.