The problem is there is no continuity. An email from an organisation that has emailed you a hundred times before looks the same as an email from somebody who has never emailed you before. Your inbox is a collection of legitimate email floating in a vast ocean of email of dubious provenance.

I think there’s a fairly straightforward way of fixing this: contact requests for email. The first email anybody sends you has an attachment that requests a token. Mail clients sort these into a “friend request” queue. When the request is accepted, the sender gets the token, and the mail gets delivered to the inbox. From that point on, the sender uses the token. Emails that use tokens can skip all the spam filters because they are known to be sent by authorised senders.

This has the effect of separating inbound email into two collections: the inbox, containing trustworthy email where you explicitly granted authorisation to the sender; and the contact request queue.

If a phisher sends you email, then it will end up in the new request queue, not your inbox. That should be a big glaring warning that it’s not a normal email from somebody you know. You would have to accept their contact request in order to even read the phishing email.

I went into more detail about the benefits of this system and how it can be implemented in this comment:

https://news.ycombinator.com/item?id=44969726

You don't need complex token arrangements for this. You can just filter emails based on their from addresses.