You are making a broad generalization and even it is based on the assumption that the page-ranking algorithm is the only possible way to do it.

I make no assumption beyond the one that if you want to index a page you need to know its address and if the author of the website is not going to give it to you because you are not Google and no known website links to it, then you have no way of finding it out. You can't build anything independently. You have to go to Google.