But if you only remember facts, can someone else pretend to be that person you remember with those facts?

No, memory is not limited to facts. There's just no visualization to go along with the recollection of faces etc. You can absolutely still recognize people correctly when looking at them.

I'm also an aphant. Recognition still happens. I'll see a person I haven't seen in years, know that I know them, then finally remember who they are much later. Even though I can't picture them after seeing them, I'll still remember who they are much later. Usually, anyway.

I get stressed out having to meet someone I haven't seen for a long time in a cafe or something. I have to scan everyone and wait for that ping of recognition to happen.

That's a fascinating question - I'm aphantasic and mildly face blind (I thought the two female leads in Disclosure Day were the same person for most of the movie and that we were jumping back and forth in her timeline). I've always assumed I'm face blind because of the aphantasia, but I know several aphantasics who have no problem recognising faces.

In answer to the question though, no - even as someone with face blindness, the faces of people I've known and loved for a long time are burned in somewhere - I just can't access that part of my brain at will.