Over time, I've learned to accept that many people -- even very clever ones -- are incapable of holding a metaphor at arm's length. Once they accept the words of a metaphor as applicable at all, the metaphor collapses entirely into literalism for them. They can no longer see that the metaphor was just a tool with inherent limitatation and boundaries.

Because the field of artificial "intelligence" is constructed around the idea of applying psychological metaphors to computational systems (a very powerful idea!) it's almost a worst case scenario for these people.

Suddenly, they're reversing the metaphors and applying computational schema to psychological processes ("aren't we really just stochastic parrots ourselves?!"); or, like here, they find themselves surprised and confused when they stumble across the natural boundaries of the metaphor experimentally.

It's because they never had sight of the boundaries in the first place and maybe never can quite see them. The words only make sense to them as literal equivalence, and so their surprise when they run into stuff like this is earnest and deep.

I don't think the boundary between "generalization" and "metaphor" is very well defined. When you go from an exemplar of 1 to 2, you're going to find all kinds of edge cases where attributes of the thing being demonstrated that had seemed to be essential turn out to not be necessary.

I think you certainly could look at LLMs as "thinking" metaphorically, but I also don't think it is necessarily only a metaphor.