Humans can explain how they arrived at the conclusion, an LLM fundamentally cannot do that since they don't remember why they picked the tokens they did, they just make up an explanation based on explanations it has seen before.