Remix.run Logo
▲ alex43578 18 hours ago

Internet search isn’t one of those. Breadth, depth, and quality of search results (ie what you want is the first result) matter. That’s only reasonably attainable at scale. Not to mention it’d be a PITA to optimize for a dozen competing crawl standards.

▲MatejKafka 14 hours ago | parent [-]

You don't need a dozen crawlers, just a dozen good search engines that have access to an existing index.