Remix.run Logo
▲ root_axis 2 hours ago

I'm not arguing that LLMs aren't useful tools for mathematicians, I'm questioning the quality of discourse that amounts to "I pasted this into the LLM and it said there are errors".

▲fasterik 2 hours ago | parent | next [-]

There's value in knowing whether a claim is true or false. Plugging things into an LLM doesn't necessarily give 100% certainty or tell us why something is true or false in a human-comprehensible way, but it can move our credences and suggest directions for investigation.

▲mtlmtlmtlmtl 30 minutes ago | parent [-]

If you're not even gonna share what the supposed errors were, the value is pretty much infinitesimal.

Surely, GPT said something more elaborate than "nuh uh".

▲jiggawatts 33 minutes ago | parent | prev | next [-]

Think of it as a (fuzzy) compiler.

Now think about complaining that someone "merely pasted my code into a compiler and said that there are errors in my code."

That's nuts, right? If a compiler -- a kind of proof engine -- finds errors, there are errors. Fix them! Don't complain about the usage of a compiler, the errors were real, they're there, they need to be fixed.

The fuzziness part is both useful and a liability, but the error rate is dropping.

Some numbers for you:

A year ago, I ran a huge legacy codebase through Gemini Pro 2.5 and it spat out several thousand bugs, but manual inspection showed that 80% were false positives and only 20% were worth fixing.

I did a similar exercise with GPT 5.6 more recently, and the hit rate was 80% valid and 20% false positive. The human developer got into an argument saying that they thought all 100% were false positives. He lost. The AI could easily produce "repro" snippets that triggered every issue (mostly security vulnerabilities, but some correctness ones too).

These days, if you're publishing anything with verifiable claims in it, and not running it through an AI first, it's the direct equivalent of uploading open source code to GitHub without bothering to check that it compiles first.

It's sloppy.

▲whimsicalism an hour ago | parent | prev [-]

classic HN isolated demand for rigor. i found it interesting