What are people actually using these hosted search providers for? For personal use I've found using searxng-ai-kit [1] (disclaimer: I'm the author) or similar tools to be far more cost effective (free!). oh-my-pi uses a similar method and will scrape search results for you out of the box, no api key or search provider needed.

[1]: https://github.com/nikvdp/searxng-ai-kit

I have built my own search engine, which is really great btw. I self-host it.

I’ve combined multiple paid search providers and some that don’t have APIs. These API-less providers I’ve written wrappers for that scrape the normal html pages and return objects from the results.

These wrapped search providers very often block the engine from getting results. Either through rate limits or by blocking my server’s IP.

I am the only user and still get blocked. Let alone if multiple people would hit let’s say DuckDuckGo from the same office IP all day or in a long research session that requires many look ups.

An obvious solution is using a proxy with automatic rotation or some sort of captcha breaking system, which is fine for an agent, but in a search engine scenario might up costing too much in latency.

Any organization that wants reliable latency, data processing guarantees (it's a big leak vector for company secrets), etc., will use such a search provider over the solutions you outlined.

Especially for chat-like use-cases (in contrast to coding), latency is a lot more critical, especially if you kick off a small single query question. On antother axis, scrape-reliabilty is important for a deep research job that will fire ~10+ web searches and ~100+ web fetches.

e.g. in our chat platform[0], we offer first party MCP servers for web search[1] and web access[2] which work decently well, but some of our customers that prioritize quality over cost, they go with dedicated providers like LinkUp[3]

[0]: https://github.com/EratoLab/erato

[1]: https://github.com/EratoLab/web-search-mcp

[2]: https://github.com/EratoLab/web-access-mcp

[3]: https://www.linkup.so/

Hosted search API from your LLM's provider is much easier to use, faster and less error prone. Also I believe that LLM may work better with its provider's search api, as it was definitely trained on it. I mean, it is just more convenient to use your provider's search.

And if we talking about LLMs, there is no particular point for search to be absolutely free if you still pay for the LLM. It would have more sense if search AND llm would be completely free.

How do you deal with fcking cloudflare blocking access to the websites ?