>They speak fluently and confidently about experiences it’s impossible for them to have.

But they're echoing these things from people who really have.

The key is to not forget that LLMs are just next-generation search engines, instead of anthropomorphizing them to be "speaking agents". The natural language IO interface is just a side effect.

Sorry to be annoying on that, but if there's one thing LLMs certainly aren't are search engines. They don't index content predictably, they lossly compress it, and furthermore in a manner that the loss cannot be quantified. If you are using them as more than a semi-random/fuzzy content production engines, you are doing it wrong (you're not alone in that, but that's besides the point)