The point from the end of the post that AI produces output that sounds correct is exactly what I try to emphasize to friends and family when explaining appropriate uses of LLMs. AI is great at tasks where sounding correct is the essence of the task (for example "change the style of this text"). Not so great when details matter and sounding correct isn't enough, which is what the author here seems to have rediscovered.

The most effective analogy I have found is comparing LLMs to theater and film actors. Everyone understands that, and the analogy offers actual predictive power. I elaborated on the idea if you're curious to read more:

https://jstrieb.github.io/posts/llm-thespians/

I wanted to check the prime factors of 1966 the other day so I googled it and it led me to https://brightchamps.com/en-us/math/numbers/factors-of-1966 , a site that seems focussed on number facts. It confidently states that prime factors of 1966 are 2, 3, 11, and 17. For fun I tried to multiply these numbers back in my head and concluded there's no way that 6 * 187 could reach 1966.

That's when I realized this site was making heavy use of AI. Sadly, lots of people are going to trust but not verify...

> “To stave off some obvious comments:

> yoUr'E PRoMPTiNg IT WRoNg!

> Am I though?”

Yes. You’re complaining that Gemini “shits the bed”, despite using 2.5 Flash (not Pro), without search or reasoning.

It’s a fact that some models are smarter than others. This is a task that requires reasoning so the article is hard to take seriously when the author uses a model optimised for speed (not intelligence), and doesn’t even turn reasoning on (nor suggest they’re even aware of it being a feature).

I asked the exact prompt to ChatGPT 5 Thinking and got an excellent answer with cited sources, all of which appears to be accurate.

They are very good at some tasks and terrible at others.

I use LLMs for language-related work (translations, grammatical explanations etc) and they are top notch in that as long as you do not ask for references to particular grammar rules. In that case they will invent non-existent references.

They are also good for tutor personas: give me jj/git/emacs commands for this situation.

But they are bad in other cases.

I started scanning books recently and wanted to crop the random stuff outside an orange sheet of paper on which the book was placed before I handed the images over to ScanTailor Advanced (STA can do this, but I wanted to keep the original images around instead of the low-quality STA version). I spent 3-5 hours with Gemini 2.5 Pro (AI Studio) trying to get it to give me a series of steps (and finally a shell script) to get this working.

And it could not do it. It mixed up GraphicsMagick and ImageMagick commands. It failed even with libvips. Finally I asked it to provide a simple shell script where I would provide four pixel distances to crop from the four edges as arguments. This one worked.

I am very surprised that people are able to write code that requires actual reasoning ability using modern LLMs.

https://dubesor.de/WashingHands

This is my personal favourite example of LLMs being stupid. It's a bit old but it's very funny that Grok is the only one that gets it..

>This is a pretty simple question to answer. Take two lists and compare them.

This continues a pattern as old as home computing: The author does not understand the task themselves, consequently "holds the computer wrong", and then blames the machine.

No "lists" were being compared. The LLM does not have a "list of TLDs" in its memory that it just refers to when you ask it. If you haven't grokked this very fundamental thing about how these LLMs work, then the problem is really, distinctly, on your end.

The training data is not automatically in the context scope, and on list tasks LLMs have nearly no way to ensure completeness due to their fundamental characteristics.

To do a task like this with LLMs, you need to use a document for your source lists or bring them directly into context, then a smart model with good prompting might zero-shot it.

But if you want any confidence in the answer, you need to use tools: “here is two lists, write a python script to find the exact matches, and return a new list with only the exact matches. Write a test dataset and verify that there are no errors, omissions, or duplicates.”

LLMs plus tools / code are amazing. LLMs on their own are a professor with an intermittent heroin problem.

https://chatgpt.com/share/68cffaab-4c14-8006-89a2-1818172e4d...

Tried on ChatGPT, seems fine.

I have used the approach where if it is more complex, then I write the code manually myself, make sure it does what I want etc. I then ask ChatGPT to have a look at it and see where the problems are. Rather than do a complete rewrite, it points out very specific problems which I can also evaluate myself. For example, identifying memory leaks and it shows me the actual changes. Where it can be done by parallel processing and what changes for that. You can look at it with experience and say, "Yes, that makes sense" and apply it as necessary. See it as a more iterative process, rather than expect AI to correctly do all the work. For trivial examples like what Terence has given, this will be very easy to code and you wouldn't expect AI. However, you could still say to AI, "Take a look at the code I have written. Can you identify how it can be done better?", and hopefully it comes back to you saying, "No, that looks pretty good to me." for such a trivial example. :-)

The other day I found that they were struggling with "find me two synonyms of 'downloading' and 'extracting' that are the same length" because I was writing a script and wanted to see if could align the next path parameter.

First there's the tokenization issue, the same old "how many R in STRAWBERRY" where they are often confidently wrong, but I also asked not to mix tense (-ing and -ed for example) and that was very hard for them.

> To be clear, I would expect a moderately intelligent teenager to be able to find two lists and compare them. If an intern gave me the same attention to detail as above, we'd be having a cosy little chat about their attitude to work.

sure, but when I expect this [1] from _any_ full time hire, my "expectations are too high from people" and "everybody has their strengths"

[1] find a list of valid html5 elements, find a list of TLDs, have an understanding of ccTLDs and gTLDs

Don't use a microwave to fry a steak then. This is an irritating post and I have plenty of skepticism towards AI. LLMs were always bad at this kind of task, simple to us humans as it may be. This post proves nothing that wasn't known for two years.

However, I do superficially agree with some of the links at the end. LLMs as they have been so far are confirmation machines and it does take skill to use them effectively. Or knowing when not to use them.

> I think it comes down to how familiar you are with the domain and its constraints. When I watch a medical drama, I have no idea if they're using realistic language. It sounds good, but real doctors probably cringe at the inaccuracies.

By now, numerous notable programmers have reported positive experiences with all forms of AI-assisted coding, which this conclusion arrogantly fails to account for.

Author seemed to have used a weak model since the strong models get the answer. They should have put more thought into it and at least provide a comparison.

As a ChatGPT user I would have reached for the thinking model for such questions. I understand if the “auto” model doesn’t pick the right model here - but confident claims from the author should be backed up by at least this much.

More generally LLMs are bad at exhaustivity: asking "give me all stuff matching a given property" almost always fails and provide at best a subset.

If possible in the context, the way to go is to ask for a piece of code processing the data to provide exhaustivity. This method have at least some chance to succeed.

Claude Opus 4.1 generated me a small web app in two minutes to find the correct answer: https://claude.ai/public/artifacts/ffbb642b-8883-4b4d-8699-d...

> Answering the question was a little tedious and subject to my tired human eyes making no mistakes

Who would do this manually? Concatenate the two lists and sort them. Use "uniq -c" to count the duplicate lines and grep to pull out the lines which occur twice. It would take a few seconds.

I see a big issue with these tools and services we call "AI".

On one hand you hear things like "AI is as smart as college student", "AI won a math competition", "AI will replace white collar workers". And so on. I'm not going to bother looking up actual references of people saying these exact things. But unless I'm completely delusional, this is the gist of what some people have been saying about AI over the past few years.

To the layperson, this sounds like a good deal. Use a free (for now) tool or pay for an advanced version to get stuff done. Simple.

But then you start scratching beneath the surface and you start hearing different stories. "No, you didn't ask it right", "No, that's a bad question because they tokenize your input", "Well, you still have to check the results", "You didn't use the right model".

Huh? How is a normal person supposed to take this stuff seriously? Now me personally, I don't have much of an issue with this stuff. I've been a developer for many, many years and I've been aware of the various developments in the field of machine learning for over 15 years. I have kind of an intuition about what I should use these systems for.

But I think the general public is misinformed about what exactly these systems are and why they're not actually intelligent. That's a problem.

https://chatgpt.com/s/t_68cffbc05ef48191996ffbaa3c6e55a7

Is this the right answer? Seems like it. I used the thinking model.

I don't ask LLM to do that I ask LLM write me a python script to do that. I am not a programmer but to me using llm to do stuff like comparing list etc is not understanding what LLM are as well as huge waste of resources

Why didn't he post the correct answer himself?

Surprisingly, really? Is anyone still surprised by that?

I had a strange AI experience the other day. I asked Kimi K2 to write a 6809 indexed operand decoder in C, and this is what it gave:

  uint16_t ea_indexed(void)
  {
      uint8_t post = *PC++;
      uint16_t base, off = 0;

      /* 1. pick base register */
      static const uint16_t *const base_tbl[4] = { &X, &Y, &U, &S };
      base = *base_tbl[(post >> 5) & 3];

      /* 2. work out the effective address */
      if ((post & 0x80) == 0) {                /* 5-bit signed offset */
          off = (int8_t)(post << 3) >> 3;
      } else if ((post & 0x60) == 0x20) {      /* 8- or 16-bit offset */
          if (post & 0x10) {                   /* 16-bit */
              off = (int16_t)fetch_be16(PC);
              PC += 2;
          } else {                             /* 8-bit */
              off = (int8_t)*PC++;
          }
      } else if ((post & 0x60) == 0x40) {      /* auto inc/dec */
          int8_t step = ((post & 0x0F) == 0x0) ? 1 :
                        ((post & 0x0F) == 0x1) ? 2 :
                        ((post & 0x0F) == 0x2) ? -1 :
                        ((post & 0x0F) == 0x3) ? -2 : 0;
          if (step > 0) base += step;          /* post-increment */
          off = step < 0 ? step : 0;           /* pre-decrement already applied */
          if (step < 0) base += step;
      } else if ((post & 0x60) == 0x60) {      /* accumulator offset */
          static const uint8_t scale[4] = {1,1,2,1};   /* A,B,D,illegal */
          uint8_t acc = (post >> 3) & 3;
          if (acc == 0) off = A;
          else if (acc == 1) off = B;
          else if (acc == 2) off = (A<<8)|B;   /* D */
          off *= scale[acc];
      } else {                                   /* 11x111xx is illegal */
          illegal();
      }

      uint16_t ea = base + off;

      /* 3. optional indirect */
      if (post & 0x10) ea = read16(ea);

      return ea;
  }

From looking at Page 150 of https://colorcomputerarchive.com/repo/Documents/Books/Motoro... it looked pretty much perfect except for the accumulator addressing. That's impressive...

Then in another chat I asked it "give a technical description of how the 6809 indexed operands are decoded" and it just can't do it. It always gets the fundamentals wrong and makes pretty much everything up. Try it yourself, doesn't have to be Kimi most other AIs get it wrong too.

My assumption is that it's learned to how to represent it in code from reading emulator sources, but hasn't quite mapped it well enough to be able to explain it in English.. or something like that.*

Once again an example of "anti-ai people are those who treat LLMs as oracles, not the pro-ai people."

> "Something that describes how an AI is convincing if you don't understand its reasoning, and close to useless if you understand its limitations."

This made me laugh. Because it's the exact opposite sentiment of anti-LLM crowd. So which is it? Is it only useful if you know what you're doing or less useful if you know what you're doing?

> "I can't wait until I can jack into the Metaverse and buy an NFT with cryptocurrency just by using an LLM! Perhaps I can view it on my 3D TV by streaming it over WIMAX? I'd better stock up on quantum computers to make sure it all works."

In the author's attempt to be a smartass, they showed themselves. It makes them sound childish. Instead of just admitting they were wrong, they make some flippant remark about cryptocurrency and NFT'S, despite having vastly different purposes and goals and successes. Just take the L.

to add: "I shouldn't have to know anything about LLMs to use them correctly" is one heck of a take, but ok.

> "I don't. I hate the way this is being sold as a universal and magical tool. The reality doesn't live up to the hype."

And I hate the way in which people will do the opposite: claim it has no uses cases. It's literally the same sentiment, but in reverse. It's just as myopic and naive. But for whatever reason, we can look at a CEO hawking it and think "They're just trying to make more money" but can't see the flipside of devs not wanting to lose their livelihoods to something. We have just as much to lose as they have to gain, but want to pretend like we're objective.

TLDR; OP used LLM models without search + reasoning and get bad results. He then concludes: Don't believe the hype.

So are people?