| ▲ | famouswaffles 16 hours ago |
| >That’s a pretty high bar, and I don’t actually believe that LLMs have closed the gap with it by all that much. They still have so many obvious tells. Frontier Labs are not interested in having LLMs being able to pass as humans. If anything, they explicitly train them not to. In many ways, this ability has regressed severely since the original GPT-3 with no instruct tuning or RL. How many 'tells' would there be really if a frontier model trained with frontier techniques is optimized to pass this test? I think this was something Turing did not quite forsee. That such machines might be created but not really care about this specific shape of the test. Regardless, i think his broader point about functional equivalence is spot on. |
|
| ▲ | bunderbunder 16 hours ago | parent [-] |
| GPT-3 might not have said “load bearing” as much, but a savvy interrogator could still catch it out nearly every time just asking dumb gotcha questions like, “How many Rs are there in strawberry?” |
| |
| ▲ | famouswaffles 16 hours ago | parent [-] | | Those are questions that are sidestepped with simply a different input paradigm than BPE tokenization. See the Byte Latent Transformer - https://arxiv.org/pdf/2412.09871 - where a similar scale byte latent model trained on the same dataset >>> a vanilla transformer on word and character manipulation tasks. For example, Llama 3 trained on 1T tokens scores 1.1% on a CUTE spelling benchamrk, while the equivalent byte latent equivalent trained on the same dataset scores 99.9%. Another example is 0.4% vs 48.7% on a Substitute Char benchmark. It all falls down to the same thing. Researchers are not optimizing for passing as a human. | | |
| ▲ | bunderbunder 16 hours ago | parent [-] | | So, sure, we can special plead the Turing test out of the picture. But if we don’t propose an alternative to take its place, we’re left right back at the same silly situation that the top level commenter was observing and that Turing was trying to move away from: enmired in a useless, meaningless argument about semantics. | | |
| ▲ | famouswaffles 15 hours ago | parent [-] | | The Turing Test itself in the form exactly envisioned is not important. You don't need an alternative 'special test'. You just need to understand the point Turing was making. That the question itself is an irrelevant one by its very nature. People that want to be enmired in meaningless semantic debates will continue to do that, no matter what test you devise. | | |
|
|
|
