Remix.run Logo
raincole an hour ago

And how many claims human experts disagree on in the exact same setting?

I'm not being snarky here. Without something to compare to the 67% number tells us nothing. And it's known that many humans disagree with human fact checkers too (see: any election around the world.)

kostaj 27 minutes ago | parent [-]

Agree. Human experts also struggle agreeing on this type of claims. The inter-annotator agreement on the verdicts on the AVeriTeC corpus across 50 organizations is κ=0.619 - substantial but well short of perfect.