> I wonder sometimes how many scams I've avoided simply by pretty much never answering my phone when someone calls unless I'm expecting a call or it's someone I know.

The answer is almost certainly greater than 0.