Something I've found surprisingly effective is telling ChatGPT to "use credible sources" - you can then watch its thinking trace and see it do things like ruling out random blogs, considering media publications with a good reputation for fact checking, and double-checking information that seems unlikely.

▲

skorniienko 4 hours ago | parent | next [-]

Hi, ran the experiment with "credible sources". Results we unexpected but not less interesting. Few things worth noting: - same search query ran minutes apart return different sources, sometimes with only 1-2 overlaps. - 16 high-credibility news outlets are rejecting agent's access to a content. (like Reuters, NYT, Guardian), but all .gov, dedicated fact-checkers are working.

Detailed report, if you are interested: https://github.com/SerhiiKorniienko/bullshit-detector/blob/m...

▲

vunderba a day ago | parent | prev | next [-]

For any LLM with web tool usage I’ll often suggest that it pull relevant papers from arXiv, with an expectation that it cite sources. I find that this helps improve accuracy and lets me go deeper into the material if I want to.

▲

westoque a day ago | parent | prev | next [-]

I don't think random blogs should be ruled out. Just because a blog is random doesn't mean its untrue.

	▲	simonw a day ago \| parent \| next [-]
		The reputation of the writer matters a lot. Recent models seem to be pretty good at evaluating author credibility as well. What's dangerous is treating information from an anonymous author as factual. You gotta check that in other ways.
	▲	bulbar a day ago \| parent \| prev [-]
		It should not be seen a credible source though. It can be, but that's hard to correctly judge on at scale.

▲

skorniienko a day ago | parent | prev | next [-]

Love it, I'll run tests for "credible sources" and will compare the results. Could be a first feature-request implemented!

▲

skorniienko a day ago | parent | prev [-]

Added this as issue so I don't forget https://github.com/SerhiiKorniienko/bullshit-detector/issues...