I think a better analogy would be conducting a marathon in a fog. If you can't see the path of the proof, how do you know it is completely true in all scenarios? If you can't see whether the AI runner ran through every part of the race, how do you know it didn't draw hallucinated shortcuts in the parts where humans can't see? Or worse, create obscurity and blow smoke to hide the shortcut section? If the same AI was to guide the last living humans to a star, because it found a path clear of danger, could you trust it to get in that ship? Or did it just forgot mentioning an asteroid belt the ship is not built to navigate? Truth is verifiable truth that multiple parties can agree upon. Can you trust with your life something you can't verify?