If human psychology worked like that, lotteries wouldn't be a thing. Nor prayer. There wouldn't be horoscopes in newspapers, nor homeopathy.

One of the various oddities going on with LLMs in particular is them being trained with feedback from users having a chance to upvote or downvote responses, or A/B test which of two is "better". This naturally leads to things which are more convincing, though this only loosely correlates to "more correct".

No shit. Why do people in this thread keep telling me that people are stupid like that's a news flash to me? The fact remains that it is stupid, and especially for educated people like the laywers/doctors/etc mentioned upthread, it's sufficiently obvious stupidity that there's no excuse. Yes, I know, that describes a lot of other stupidity. Much of our history as a species is inexcusable.

Edit: though I should be clear: people demonstrably do often learn to discount obviously unreliable sources. Not all the time, but pretty often in the easily verifiable cases, especially where they don't have a major emotional stake.