I think they're recognising patterns, which is not the same thing.

Abstractions are stable, they're explicit in their domains, good abstractions cross multiple domains, and they typically come with a symbolic algebra of available operations.

Math is made of abstractions.

Patterns are a weaker form of cognition. They're implicit, heavily context-dependent, and there's no algebra. You have to poke at them crudely in the hope you can make them do something useful.

Using LLMs feels more like the latter than the former.

If LLMs were generating true abstractions they'd be finding meta-descriptions for code and language and making them accessible directly.

AGI - or ASI - may be be able to do that some day, but it's not doing that now.