I think the article gives a much more plausible explanation for the demise of the Turing Test: the jagged frontier. In the past being able to write convincingly well seemed like a good overall proxy for cognitive ability. It turns out LLMs are excellent at spitting out reasonable sounding text, and great at producing certain types of writing, but are still terrible at many writing tasks that rely on cognitive ability.

Humans don't need to cast about for obscure cases where they are smarter than an LLM, there are an endless supply of examples. It's simply the case that the Turing Test tells us very little about the relative strengths and weaknesses of the current AI capabilities.

The turing test basically subsumes all tests that can be text-encoded, no? Like if you feel that LLMs are abnormally bad at a kind of writing like an All Souls essay, you just ask the other chair to write you such an essay as one of your questions.

To be clear, I'm not aware of anyone actually running any serious turing tests today because it's very expensive and tedious. There's one being passed around where each conversation is only 4(!) little SMS-sized messages long per side, and chat gpt gets judged to be the human side twice as often as the actual human.