The classic God of the Gaps - we don't know how human brains think, so what LLMs do must be it!

I'm not saying that LLMs do anything, just that it's rich to confidently say they don't do something when we don't even understand how humans do it.

It's like we're pretending cognition is a solved problem so we can make grand claims about what LLM's aren't really doing.