Feed the LLM output into a “deterministic” verifier, problem solved. That’s how LLMs verify their new mathematical proofs with lean.