A little over 15 years ago you could index the web with a small cluster. I remember people doing doing it with Cassandra or Elasticsearch. I'm sure you'd need a much bigger cluster, but outside video and images I imagine it's still doable even for a small organization, especially if you're filtering out content farms. Plus, there are many organizations interested in having access to an index, and I'm pretty more than a few currently running their own index and selling to analytics firms.

▲

piskov 9 hours ago | parent [-]

Index is one thing, great search over it is another.

A competitive, general-purpose web search engine with its own full index is _brutally_ hard and expensive.

This is the reason there are only a few world-class like russian yandex, chinese baidu (to not state the obvious names like google).

	▲	tensor 2 hours ago \| parent [-]
		They already have world class search technology. In terms of indexing they only need to index the most important content. There is so much slop now and it doesn't matter at all if that is indexed. IMO their strategy of focusing on smaller sites with human curated content is correct. They can make some deals to index some of the big walled gardens and that's pretty much world class right now. What do you think a site like Google is giving you these days? They are explicitly bad at indexing the small web. Their search technology is not better than Kagi, and made worse by ads and LLM ad bias. So what is this big "world class" thing they do that can't be replicated? The web is not the same place it was years ago. Indexing all the slop and scams and ads is not useful to me as a consumer.