This is getting to be possibly the most irritating thing I've seen on Hacker News since registering here. Every thread about a limitation of LLMs being immediately rebuked with "humans do that too."

It's a continuous object lesson in missing the point. A similar thing happened a few hours ago when an article was posted about a researcher who posted a fake paper about a fake disease to a pre-print server that LLMs picked up via RAG, telling people with vague symptoms that they had this non-existent disease. Lo and behold, commenters go in immediately saying "I'd be fooled too because I trust pre-print medical research." Except the article itself was intentionally ridiculous, opening by telling you it was fake, using obviously fake names, fictional characters from popular television. The only reason it fooled humans on Hacker News is because they don't bother reading the articles and respond only to headlines.

It's just like your code examples. Humans fail because we're lazy. Just like all animals, we have a strong instinct to preserve energy and expend effort only when provoked by fear, desire, or external coercion. The easiest possible code to write that seems to work on a single happy path using stupid workarounds is deemed good enough and allowed through. If your true purpose on a web discussion board is to bloviate and prove how smart you are rather than learn anything, why bother actually reading anything? The faster you comment, the better chance you have of getting noticed and upvoted anyway.

Humans are not actually stupid. We can write great code. We can read an obviously fake paper and understand that it's fake. We know how hierarchy of evidence and trust works if we bother to try. We're just incredibly lazy. LLMs are not lazy. Unlike animals, they have no idea how much energy they're using and don't care. Their human slaves will move heaven and earth and reallocate entire sectors of their national economies and land use policies to feed them as much as they will ever need. LLMs, however, do have far more concrete cognitive limitations brought about by the way they are trained without any grounding in hierarchy of evidence or the factual accuracy of the text the ingest. We've erected quite a bit of ingenious scaffolding with various forms of augmented context, input pre-processing, post-training model fine tuning, and whatever the heck else these brilliant human engineers are doing to create the latest generation of state of the art agents, but the models underneath still have this limitation.

Do we need more? Can the scaffolding alone compensate sufficiently to produce true genius at the level of a human who is actually motivated and trying? I have no idea. Maybe, maybe not, but it's really irritating that we can't even discuss the topic because it immediately drops into the tarpit of "well, you too." It's the discourse of toddlers. Can't we do better than this?