> If the LLM output is more effective than a human at problem solving, which I think we can all agree requires intelligence

Your premise is wrong.

Unless you want to claim that the distant cause by way of the training data is us, but that's exactly the conclusion you're trying to avoid. After all, we put the patterns in the training data, which means we already did the upfront intellectual work for the LLM.