I'm not convinced—and certainly don't find it obvious—that this couldn't ultimately also be how we reason as humans.

It's clear that there's an enormous amount of leverage built into language-as-practiced that one can use to engage in a broad spectrum of reasoning, from the extremely fallible off-the-cuff conclusion to the deeply-considered and rigorous proof. How do we know this leverage is built into language-as-practiced? Because LLMs can do a broad swath of it.

But how do we know we're not doing something similar?

I don't think we can assume that we're not simply by observing that we're not digital and we don't use matrix multiplication. Why immediately dismiss the possibility that there might be a similar, but biomechanical, computation at play in our heads that plays in the same space of vectors?

humans reasoned before language but lacked the ability to store and transmit it. Later advanced humans developed abstract reasoning once lingustistic library became sufficiently description of reality. But this is not how we reason from first princples.

Language is one of our tools we developed to STORE reasoning, not create it. LLMs excel at uncompressing and interpreting that stored reasoning.