I get a lot of emails that I want to get but never reply to. I would not want to have to remember to whitelist all of those.

Yeah fair, then you could set it higher, even 100. Or default it off.