Uh yeah... I would not use that tool. A tool which doesn't do its job randomly is useless.

Generally I’d ditch that tool because it doesn’t work. A calculator is supposed to calculate. If it can’t reliably calculate, then it’s not a functioning tool and I am tired of people insisting it is functioning properly.

LLM’s simply aren’t good enough for all the use cases some people insist they are. They’re powerful tools that have been far too broadly applied and there’s too much money and too many reputations being put on the line to acknowledge the obvious limitations. Frankly I’m sick of it.

I had somebody on HN a few months ago insist to me that because we value art and fiction, LLM’s being wrong when we need them to be correct (in ways that are also not always easy to identify) was desirable. I don’t even know what to do with that kind of logic other than chalk it up as trolling. I don’t want my computer to trick me into false solutions.

> It's not like these are new issues.

Exactly, that's why not verifying the output is even less defensible now than it ever has been - especially for professional scientists who are responsible for the quality of their own work.

> Worse still, whenever they're brought up users are blamed for "holding it wrong", or for misunderstanding how the tools work. I don't care. An "artificial intelligence" shouldn't be plagued by these issues.

My feelings exactly, but you’re articulating it better than I typically do ha