I have built my own search engine, which is really great btw. I self-host it.

I’ve combined multiple paid search providers and some that don’t have APIs. These API-less providers I’ve written wrappers for that scrape the normal html pages and return objects from the results.

These wrapped search providers very often block the engine from getting results. Either through rate limits or by blocking my server’s IP.

I am the only user and still get blocked. Let alone if multiple people would hit let’s say DuckDuckGo from the same office IP all day or in a long research session that requires many look ups.

An obvious solution is using a proxy with automatic rotation or some sort of captcha breaking system, which is fine for an agent, but in a search engine scenario might up costing too much in latency.