This is a fair comment, although I would add that is not my expectation personally.

I think nondeterminism does not have to be the same as non-coherency - i.e. just because something is randomly sampled does not mean the result has to be incoherent or inconsistent.

Also, if we speak purely about LLM based on how they are implemented now, I feel that is different than speaking about artificial intelligence. The field of AI is much more than just an LLM by itself, and the promise of these companies is not just LLM, whether the underlying models are limited to that technology or not.

FWIW, I have built rule based expert systems, used logic based reasoning systems like NASA CLIPS or rete-algorithm based systems, mathematical/symbolic solvers, written plenty of terrible case/conditional logic in programming languages, worked with ML in its infancy and now worked in AI/LLMs - I give this context only to clarify that I understand what an LLM is and isn't.

With all that said, LLMs have allowed humanity to make advances, at great cost to society (IMHO), and I'd hate to see the opportunity be wasted.

There is plenty of room past "attention is all you need" still to do incredible work, especially at the crossroads between deterministic and nondeterministic behaviors.