As an example, if you call your bank to report a lost credit card, and that you'd like it shipped to a different address than the one you registered with them, they'll ask you for the last 4 digits of your SSN.

So yeah, someone who knows (name, SSN) or especially (name, address, phone, SSN) can do a lot of harm.