>LLMs-as-AI can only give us probabilistic outputs based on inputs

I am not completely sure what you are saying here, but it sounds like a variation of the "it's just a stochastic parrot" argument, which is reductionist. The human brain is also just a bunch neurons firing.