There's another element to this that I almost never see discussed. Ideas are not facts. Neither LLMs nor humans can generate new knowledge, as opposed to ideas, by thinking alone. Physical investigation and experimentation is necessary.

The exception being pure mathematics since it exists solely in the realm of ideas. I'm willing to call that knowledge, but it's still a distinction, the old analytic/synthetic dichotomy of Kant.