I think we need to disqualify humans as well. Their brains have been shown to operate on probabilistic chemical interactions and even quantum effects.

That doesn’t disqualify humans. It highlights the difference I am talking about.

Those chemical interactions and quantum effects lead to emergent properties like judgment, experience, context, accountability, and an understanding of consequences. Those are not properties that LLMs possess, regardless of how useful their output can be.

That is not to say that, in the future, LLMs won’t be used as part of other systems that add some of those properties. But that is not what we have today, or what can be seen in the foreseeable near future.