Funny you bring this up.

Back in the day when I was like 15 and DC++ was still a thing, I used to browse people's shared folders. One day I came across a file called "the paradox of false positive". It was a 1 pager that described how a machine which is 99.9% accurate at identifying terrorists would be completely useless due to this false positive base rate fallacy you're describing.

It really stuck with me throughout the years. It's kind o remarkable how even a 99.9% accurate heuristic is insufficient at scale.

Which begs the question: lets assume the intentions are pure (which we know they're not but lets be generous), what other options are there when 99.9% heuristic is not good enough? how do you design systems when they're guaranteed to fail as they scale up?

edit: and what do you know, I just saw this as I scrolled down on HN https://news.ycombinator.com/item?id=48816959

The intuition I've built is that you can't talk about a false positive rate being high or low on its own - it's always relative to the actual occurrence rate of positives in the tested population. E.g. if there's a 1 in 10000 risk of a false positive, but real positives also are only 1 out of 10000 tested cases, then a positive case will have a 50/50 chance of being a false positive (because for every 10000 tests, you'll have on average one false positive and one real positive). So a false positive rate can only be said to be low if it's significantly lower than the real occurrence rate of positives.