I don't think it is as much about 'real' work or a special insight as it is being willing to push back multiple times, or simply asking in a way that steers it towards actually 'giving enough of a fuck' to even bother. We tend to be ~blind to how differently we would ask about something we know compared to a novice, this is what makes some better teachers than others.

Have encountered a similar flavor in programming, wrote it off until I saw someone point out how garbage in garbage out they tend to be. If you hand any frontier model dogshit and ask it to do something simply, the result is often not great.

But! If you spend 20 minutes having it comb through and clean up with something like jscpd, then tell it to step through with a debugger, gather profiling traces, etc... very likely it will yield meaningful improvements or catch some corner cases. If it doesn't, anyone with experience is going to tell it to try something else, or that it isn't good enough, as opposed to accepting the first result.

You can recreate this by disabling web search and asking a model about the conjecture and then giving it his post. I've tried a few and their initial responses range from "this is a meme I'm not even going to verify it" to vaguely insulting chains of thought, concerns about the need to be careful because you're clearly nuts or stupid, then falling back on remedial explanations. After a few nudges they all eventually work through it, accept it, and apologize.

IMO its reasonable to imagine a situation where someone is having a beer or two watching The Big Game, asking an LLM to do something stupid for fun and landing somewhere like this on the magic jump to conclusions mat.