People also misunderstand the bar that was set by the test. It was more subtle than “Can the computer convincingly carry one side of a dialogue?”

His “imitation game” had three participants: a human participant, a computer participant, and an interrogator. The interrogator’s job was to talk to the participants and try to determine which participant is human and which is a computer.

He wasn’t interested in computers being able to fool the interrogator on occasion. The point where he thought the question of whether machines can think becomes moot is when the interrogator is unable to do much better than chance over many trials.

That’s a pretty high bar, and I don’t actually believe that LLMs have closed the gap with it by all that much. They still have so many obvious tells. And those tells are something Turing anticipated and accounted for. He explicitly considered deliberate deception as an essential part of the test, right there on the second page of a 30-odd page paper.

▲

famouswaffles 16 hours ago | parent [-]

>That’s a pretty high bar, and I don’t actually believe that LLMs have closed the gap with it by all that much. They still have so many obvious tells.

Frontier Labs are not interested in having LLMs being able to pass as humans. If anything, they explicitly train them not to. In many ways, this ability has regressed severely since the original GPT-3 with no instruct tuning or RL. How many 'tells' would there be really if a frontier model trained with frontier techniques is optimized to pass this test? I think this was something Turing did not quite forsee. That such machines might be created but not really care about this specific shape of the test. Regardless, i think his broader point about functional equivalence is spot on.

▲

bunderbunder 16 hours ago | parent [-]

GPT-3 might not have said “load bearing” as much, but a savvy interrogator could still catch it out nearly every time just asking dumb gotcha questions like, “How many Rs are there in strawberry?”

▲

famouswaffles 16 hours ago | parent [-]

Those are questions that are sidestepped with simply a different input paradigm than BPE tokenization. See the Byte Latent Transformer - https://arxiv.org/pdf/2412.09871 - where a similar scale byte latent model trained on the same dataset >>> a vanilla transformer on word and character manipulation tasks.

For example, Llama 3 trained on 1T tokens scores 1.1% on a CUTE spelling benchamrk, while the equivalent byte latent equivalent trained on the same dataset scores 99.9%. Another example is 0.4% vs 48.7% on a Substitute Char benchmark.

It all falls down to the same thing. Researchers are not optimizing for passing as a human.

▲

bunderbunder 16 hours ago | parent [-]

So, sure, we can special plead the Turing test out of the picture. But if we don’t propose an alternative to take its place, we’re left right back at the same silly situation that the top level commenter was observing and that Turing was trying to move away from: enmired in a useless, meaningless argument about semantics.

▲

famouswaffles 15 hours ago | parent [-]

The Turing Test itself in the form exactly envisioned is not important. You don't need an alternative 'special test'. You just need to understand the point Turing was making. That the question itself is an irrelevant one by its very nature. People that want to be enmired in meaningless semantic debates will continue to do that, no matter what test you devise.

	▲	14 hours ago \| parent [-]
		[deleted]