Believe it or don't, most mailservers actually are set up this way, but it's completely invisible to the user: https://en.wikipedia.org/wiki/Greylisting_(email)

Greylisting is not perfect, not even close, but it allows servers to reject huge amounts of garbage mail while allowing new mail from unexpected parties without user intervention.

It is similar but not the same thing. It does not have anything to do with the email being legit (non spam) or not. It only catches emails that do not do what they supposed to do when they are deferred for a little bit. They are supposed to try again later. (A super nice feature of email which protects you from losing all incoming emails when your server is down for a short time). The reasoning here is hopefully a spam script or software will not implement the standard properly or will skip apparently broken destinations.

What parent wants is also possible and I remember people doing it as far back as the early 2000s. You would basically manually whitelist an email or have them bypass the list (once or always) with something like a key word or token sort of like a captcha which can be shared off band.