Yeah it's a trick question, the human error rate for it was about 30% (higher depending on the country).

The thing there though is that, if a human were given time to think about it, they'd probably go "hang on a minute", and with the LLMs that didn't seem to happen. They just kept confidently reasoning down the absurd path.

That reminds me, I recently had an AI write a ton of tests proving the "correctness" of a feature it had implemented completely backwards. (I noted that if I had been using a language that required formal proofs, that wouldn't have helped either: it would have just provided a formal proof for the absurd implementation!)

It's one example that points out a major (possibly fundamental) flaw. I can point to prompt injection as another example. There are tons more if you're interested.

Are you actually claiming LLMs operate based on human-like intelligence?

I never understood what the walk to car wash thing was supposed to prove. Was it supposed to be something to blow normies' minds with on social media? Woah dude, so like, chat gpt is not actually smart? That's crazy dude.

That experiment "proved" that LLMs are statistical text generators without a concept of meanings of words. Which is the same thing as "proving" that there aren't a million tiny humans inside your laptop doing the CPU's work by hand.

If anything a large proportion of stories about AI in sci fi is about AI failing to understand humans in various ways.

> Or you can keep calling them stochastic parrots as they solve decades-old open problems.

I didn't use that phrase at all. But computers calculated digits of π to trillions of digits. With a chat interface for a Python math program would look like the most impressive math genius if you took it back a few decades.

> The real question is how useful they are...

That's not the "real question" but an entirely different question that is easily answered. Nothing I wrote suggested they're not incredibly useful.

> Data from Star Trek TNG failing to understand figures of speech.

These are just little instances of bad writing. Data is very much an attempt at displaying a human-like intelligence.

> The real question is how useful they are

No, it is not.

> and the answer "not at all" increasingly requires flat-earth levels of denial.

No, it does not.

For me, after ~25 years in the skeptics movement, I think the parallels with supplementary, complementary and alternative medicine are most useful.

I choose that term intentionally: its initials are S.C.A.M. and that's exactly what it is. As Tim Minchin and Alan Kay both noted, "we have a special term for alternative medicine that's been tested and shown to work. It's called 'medicine'."

If it worked, it'd be normal standard clinical medicine. But it doesn't work, and so it isn't.

And yet, SCAM is a multi-billion-dollar industry. People have ostensibly official qualifications like "ND", for "naturopathic doctor", even though that person is not a doctor and can't make you better from any kind of illness at all. Colleges teach it, millions use it, and yet, it does not work.

Which means we need to ask:

1. What does "It works! It's useful!" really mean?

2. How do we know it does not in fact work?

As a handy example, let's look at homeopathy.

Here's a quick list of things widely believed...

* It's traditional. It isn't. It was invented by Samuel Hahnemann in 1796. * It's a kind of herbal medicine. It isn't. One widely-used ingredient is duck's liver ("Oscillococcinum"). Ducks are not herbs and neither are their livers. * It's been proved to work. It hasn't.

We can go through the principles and prove it doesn't work even without going into a laboratory.

The principle is, "like cures like." A substance that causes symptoms like a given disease can treat that disease.

Fact: they can't.

Then we make that substance stronger by successive, succussive dilution.

Fact: it doesn't. That's why we say things are "watered down".

Succussive: you have to mix the diluted substance by banging the bottle against a copy of Hahnemann's book. Dude knew how to make money.

Fact: Dilution does not work.

That's why we call things "watered down." It makes them weaker.

Sufficiently high dilutions can be shown by statistics to have not a single molecule of the substance left, but that's OK because "water has a memory".

Fact: water does not have a memory.

We know from the principles it cannot work.

Relevance to AI: we know how the transformer algorithm works. It cannot think. Adding a few feedback loops for more plausible, but much more computationally expensive, answers does not miraculously add thinking, any more than banging a test tube of water and duck's liver magically mixes it better.

But people believe it, so it's been tested. It doesn't work. It doesn't work on people, or in vivo meaning when tested on animals, or in vitro meaning when tested in the lab on cell culture, or in silico which means in computational simulation.

*BUT!*

Most people get better from most things. This is called "reversion to the mean" and if it weren't so the first cold would have wiped out the cavemen.

What it can do, like all SCAM treatment, is make people feel better.

Being treated by a nice friendly doctor makes people feel better. It does not make them better -- it is only a state of mind.

That can sometimes marginally help gravely ill people rally, but only very rarely.

There is also the placebo effect, also much misunderstood.

This makes someone FEEL as if they'd had medicine if they think they've had medicine.

They do not get better. They just feel better for a bit. If they are ill, they remain ill. If they are dying, they still die.

But it might hurt less.

The placebo effect is very strong. Medicine from a person in a white coat works better than form the same person in street clothes.

Very big pills work better than smaller ones... but very small pills work better still, as a tiny pill suggests to people it's a very strong drug.

This is what "But AI works!" really means.

It makes people think they're doing less work -- in tests, they in fact do more, checking and fixing. Unless they don't check or fix, in which case, they are irresponsible fools.

It makes people think it can do amazing things because it can find prior art in its corpus they couldn't find -- or didn't look for, or know how to search for.

It does not save the need for skills.

Experienced practitioners can front-load the work with really detailed prompts which cover exceptions, edge cases, and things that novices don't know about. But the novices don't know that they don't know. (It enhances the illusion of competence. It helps the skilled more than it helps the unskilled, but neither realises, and it prevents the unskilled learning by trial and error. It reduces the supply of skilled workers.)

The reason AI works is the reason that people see the face of Jesus in slices of toast, as someone said recently.

> There are so many other sites. So many others. Why are you here?

I've been here since 2007 when HN launched.

You're confused about my objection. I don't like the term "AI" but I love the technology as much as almost anyone.

Is this website called AI Faithful News?

That's how it works though. The moment we have "AI" and see something working, it immediately ceases to be magic because "it's just a program after all." Aligning on a true definition of Artificial Intelligence is a very vexing problem.

It can however select a girls school as a military target, and did.

It has, but I think in both directions.

It used to be assumed that playing chess would require the same level of general purpose problem solving cognitive skills that the best chess players possess. But of course a Chess grandmaster that spend a few minutes learning Go can beat a Chess AI at Go with no trouble at all, because a chess AI is incapable of making effective moves in Go at all. Clearly those expectations were incorrect. Pointing that out isn't revisionism.

On the other hand, intelligence is an incredibly broad term. About as broad as a term can get. Arguably Eliza, or an Excel macro has some degree of decision making ability in some sense, it's just unbelievably primitive.

So, we need to be clearer what we mean by intelligence. We're learning that as we go along. At least now we have a few more bits of the map between us and an IF statement visible to us.

> When Deep Blue won at chess, people didn't revise their estimates of AI capabilities upwards. They revised their estimates of how much intelligence is required to play chess at world level downwards, by a lot. Surely playing chess must have never required any intelligence in the first place!

You are wrong, many did temporarily revise their estimates of AI capabilities upwards, but then 10 years later they realized they were wrong and adjusted chess downward as you say.

We have seen that pattern over and over.

"But it's not really doing arithmetic," he mumbled to himself, as he punched the numbers into his Busicom LE-120A.