> computers could count the Rs in strawberry since vacuum tubes. that measure is irrelevant.
I don't think it's irrelevant but perhaps not in the way you're assuming. When assessing AGI I'm not evaluating counting characters or even the execution of math operators at any scale or speed. As you observe, computer software from Regex to spreadsheets and Mathematica already handle that well. But AGI isn't about what computers can do, it's specifically about what LLMs can do. Especially relative to things which are have been, until now, uniquely human capabilities like understanding nuanced context coming up with novel solutions to new problems not based on prior knowledge.
Most definitions of AGI start from a baseline that already assumes easily passing a Turing test and doing anything via text response that a high school graduate could. I ding modern LLMs for failing to count but for failing to intuitively understand the nuanced context of the problem posed. I fully understand that the proximate reason LLMs fail letter counting is that they operate at the token level.
The only reason LLMs get strawberry and blueberry correct now is that they have those words on their pre-training 'cheat sheet'. However, the fundamental weakness in the way LLM intelligence works which leads to this failure mode still hasn't been addressed. Even when the frontier labs add "recognize any sub-token counting question and write a Python script" to the training 'cheat sheet' so LLMs always pass that test... to me they still have a weakness that they would fail without that cheat sheet and therefore cannot fully replicate human-like intelligence. And it's not just about letter-counting. That gap and others like it lead to many other kinds of non-human brittleness in LLM problem solving. Those are the classes of reasoning, intuition and insight that the ARC-AGI series has been trying to queue up as targets. Not to show how bad LLMs are but to help them be great in all these counter-intuitive edge cases
