Bigger search engines have a lot of custom integrations that are not just crawling the clear web. You have a lot of benefits from collecting data for a longer time in this market.

I've often switched between Google and DDG but always came back to Google as in direct comparisons I was always not finding the right results in DDG. Kagi is the first time that I have not switched back a single time as nothing changed negatively in the quality and quantity of the results.

It's a noble goal to have your own search index, but it's duplicating a lot of work that others with much more resources already do well.

In practice it is no longer actually possible to create a new search index with anywhere near the breadth of Google, regardless of the amount of resources you have available. Google (and to a lesser extant the couple of other historic engines) are entrenched enough to be allowed access by webmasters in robots.txt. And in the new era of mass abusive scraping by LLM companies, there's no longer the option to ignore robots.txt and scrape anyway, as sites do anything and everything they can to block any and all scrapers besides those already firmly entrenched.