It seems unfair, yeah.

But the main way to know is that companies never call you these days. No company should have and few do have a workflow where an employee will call you "cold" with a problem. Instead, companies email, snail mail or text about a problem and you call them back.

But if someone somehow sounds legit, you ask for the official number and whatever info is need to identify your supposed problem. Then go to the website and verify. Call the number at the website (that you find from your own search, not the caller's info) and then have them verify you.