One way to think about that is time required:

If earth is about 4 billion years old, but it takes say 400 trillion years for natural processes to produce this chemistry, then it happened out there not here.

This was a key reason why Hoyle preferred a steady state model of the universe — the part of the universe we inhabit needs to be very, very old for this stuff to work out, according to his thinking. A minority opinion, for sure, his rejection of the Big Bang model and timelines lost him a lot of respect among his peers. And his ideas could be wrong, I’m just pointing out that historically panspermia proponents have taken this position as to “why not here”.