The problem is that LLM output looks like a human conversation. People believe it.

Which is more believable?

“The sky is filled with a downpour of squealing pigs. Would you like me to suggest the best type of umbrella?”

“Sky pigs squealing”

People just don't like to be played for fools. Perhaps us giving into this is progress? I'd give a big ol' "fuck you" to anyone who claims it is, but I'm also pretty old.

I am not sure I would even say "believe", I would think of it more as short-circuiting our critical thinking. I think it taps into something at the core of our tribal instincts. It was famously present in even basic systems like Eliza. And it's not just machines... The same tricks are used by conmen, politicians, and psychopaths, which is more negative than I intend. Even with good intentions and positive outcomes, I feel we need to remember that we drive it, not the other way around.