It seems like this is the type of thing that LLMs would actually excel at though: find a list of citations and claims in this paper, do the cited works support the claims?

▲

bryanrasmussen a day ago | parent [-]

sure, except when they hallucinate that the cited works support the claims when they do not. At which point you're back at needing to read the cited works to see if they support the claims.

	▲	mike_hearn 16 hours ago \| parent \| next [-]
		Sometimes this kind of problem can be fixed by adjusting the prompt. You don't say "here's a paper, find me invalid citations". You put less pressure on the model by chunking the text into sentences or paragraphs, extracting the citations for that chunk, and presenting both with a prompt like: The following claim may be evidenced by the text of the article that follows. Please invoke the found_claim tool with a list of the specific sentence(s) in the text that support the claim, or an empty list indicating you could not find support for it in the text. In other words you make it a needle-in-a-haystack problem, which models are much better at.
	▲	BHSPitMonkey a day ago \| parent \| prev [-]
		You don't just accept the review as-is, though; You prompt it to be a skeptic and find a handful of specific examples of claims that are worth extra attention from a qualified human. Unfortunately, this probably results in lazy humans _only_ reading the automated flagged areas critically and neglecting everything else, but hey—at least it might keep a little more garbage out?