Most likely it is, the issue then becomes being able to store and afford the storage for all the files.

Sure, and that's not easy, but it's a lot easier than having to crawl the entire public Internet yourself.