I can imagine that one of the aspects of being social creatures is that we are drawn to the entropy/surprise factor of natural language produced by a brain that isn't our own. Deriving a shared understanding without being able to precisely model the language output of someone else is like, the fundamental substance of conversation.

Maybe that's the key ingredient that's missing from LLM-generated text. Even when there is actual content there, there's this lack of surprise / uncertainty in its delivery that we require to stay engaged. We may not know what will be communicated, but knowing exactly how it will be communicated just kills our innate interest.

This presumes an LLM has something to say. The reason the text doesnt make much sense, is because it doesnt. It isnt "on topic" because it has no topic to be on. Sentences go on, are joined irrelevantly, and so on because theyre sampled from a distribution of "possibly relevant sentences". They aren't being addressed to a reader (whose mind is being modelled), on a topic the writer wishes to express.

Communication expresses an understanding. LLMs arent communicating.

LLMs generate code well enough. Code has no natural language semantics. Programming's condition of satisfaction is that the machine executes correctly. The condition of satisfaction of communication is that the communication expresses what there is to be communicated.

In the limit where an LLM is trained to repoduce an exmplar of communication, it seems as if it does so. In that limit, LLMs can reporduce anything.

If frontier labs wanted to improve "LLMs for communication" they would pay for reasoning traces to be produced which reproduced the thought process of writers/speakers thinking about their audiences. Rarely do such traces exist in prior datasets .