What’s interesting here is that the model isn’t really “lying” — it’s just amplifying whatever retrieval hands it.

Most RAG pipelines retrieve and concatenate, but they don’t ask “how trustworthy is this source?” or “do multiple independent sources corroborate this claim?”

Without some notion of source reliability or cross-verification, confident synthesis of fiction is almost guaranteed.

Has anyone seen a production system that actually does claim-level verification before generation?

▲

cor_NEEL_ius 3 hours ago | parent | next [-]

The scarier version of this problem is what I've been calling "zombie stats" - numbers that get cited across dozens of sources but have no traceable primary origin.

We recently tested 6 AI presentation tools with the same prompt and fact-checked every claim. Multiple tools independently produced the stat "54% higher test scores" when discussing AI in education. Sounds legit. Widely cited online. But when you try to trace it back to an actual study - there's nothing. No paper, no researcher, no methodology.

The convergence actually makes it worse. If three independent tools all say the same number, your instinct is "must be real." But it just means they all trained on the same bad data.

To your question about claim-level verification: the closest I've seen is attaching source URLs to each claim at generation time, so the human can click through and check. Not automated verification, but at least it makes the verification possible rather than requiring you to Google every stat yourself. The gap between "here's a confident number" and "here's a confident number, and here's where it came from" is enormous in practice.

▲

rco8786 3 hours ago | parent | prev [-]

> Has anyone seen a production system that actually does claim-level verification before generation?

"Claim level" no, but search engines have been scoring sources on reliability and authority for decades now.

	▲	amabito 3 hours ago \| parent [-]
		Right — search engines have long had authority scoring, link graphs, freshness signals, etc. The interesting gap is that retrieval systems used in LLM pipelines often don't inherit those signals in a structured way. They fetch documents, but the model sees text, not provenance metadata or confidence scores. So even if the ranking system “knows” a source is weak, that signal doesn’t necessarily survive into generation. Maybe the harder problem isn’t retrieval, but how to propagate source trust signals all the way into the claim itself.