> we really need this information to even pretend to evaluate the claims of stuff like mathematical breakthroughs

Why?

Either the proof is correct, or it isn't, right?

And it either produces them reliably or not, right?

Like, even if it's reasoning is completely wrong, and it's only producing correct answers 10% of the time, that's still an astounding amount above baseline and a useful tool.

Humans have inaccurate thinking all the time, and are also pretty hopelessly opaque. "It came to me in a dream" is a major plot point in the history of math. I'd still trust Ramanujan more than most mathematicians, since he got the right answer.