How errors add depend upon how you treat them.

Two independent task? You have a 95% chance of each task being correct, so total success is .95^2 = 90.25% correct.

Having two AIs verify the same task with each being 95% correct? That's a 5% error per each, so 1-(.05^2) = 99.75% correct.

That is assuming fully independent error rates, which is the real issue. If it is pure hallucinations, then that is an okay assumption, but if there are issues in the underlying prompt/data causing the error rates, then it becomes a very dangerous assumption and I think more human involvement is recommended to better understand and resolve the underlying data problem.

The real interesting bit is that more AI's mean the 2 in the above equations gets bigger, so the error rate gets worse or better depending upon which equation is in play. So if you are in an in-between landscape where you aren't sure there is independence or not (or how much independence between errors), it also becomes unclear if more AI makes the error rate better or worse.

This. The math of coincident errors is why we can get reliable AI, but it really is a matter of how the errors manifest and how they are detected.

I’ve written a bit about this and my approach is based on clean room and N-versions.

Upvoted for real math