| ▲ | 01jonny01 7 hours ago |
| Google quietly announced that Programmable Search (ex-Custom Search) won’t allow new engines to “search the entire web” anymore. New engines are capped at searching up to 50 domains, and existing full-web engines have until Jan 1, 2027 to transition. If you actually need whole-web search, Google now points you to an “interest form” for enterprise solutions (Vertex AI Search etc.), with no public pricing and no guarantee they’ll even reply. This seems like it effectively ends the era of indie / niche search engines being able to build on Google’s index. Anything that looks like general web search is getting pushed behind enterprise gates. I haven’t seen much discussion about this yet, but for anyone who built a small search product on Programmable Search, this feels like a pretty big shift. Curious if others here are affected or already planning alternatives. UPDATE: I logged into Programmable Search and the message is even more explicit: Full web search via the "Search the entire web" feature will be discontinued within the next year. Please update your search engine to specify specific sites to search. With this link: https://support.google.com/programmable-search/answer/123971... |
|
| ▲ | zitterbewegung an hour ago | parent | next [-] |
| I know that duckduckgo uses Microsoft Bing Custom search and honestly it is a much more robust system since you don't have to worry about Google axing it. https://www.customsearch.ai |
| |
| ▲ | embedding-shape 8 minutes ago | parent [-] | | Instead you worry about Microsoft axing it? Sure, it might take 3 years instead of 6 months, and the shutdown period would be 1 year instead of 1 month, but hardly either are long-term solutions. |
|
|
| ▲ | throwaway_20357 7 hours ago | parent | prev | next [-] |
| What are some of the niche search engines build on Google's index affected by this? |
| |
| ▲ | doublerabbit 6 hours ago | parent [-] | | Kagi | | |
| ▲ | nemosaltat 6 hours ago | parent | next [-] | | > Kagi
This seems to be true, but more indirectly. From Kagi’s blog [0] which is a follow up to a Kagi blog post from last year [1]. [0]> Google: Google does not offer a public search API. The only available path is an ad-syndication bundle with no changes to result presentation - the model Startpage uses. Ad syndication is a non-starter for Kagi’s ad-free subscription model.[^1] [0]> The current interim approach
(current as of Jan 21, 2026) [0]> Because direct licensing isn’t available to us on compatible terms, we - like many others - use third-party API providers for SERP-style results (SERP meaning search engine results page). These providers serve major enterprises (according to their websites) including Nvidia, Adobe, Samsung, Stanford, DeepMind, Uber, and the United Nations. I’m an avid Kagi user, and it seems like Kagi and some other notable interested parties have _already_ been unable to do get what they want/need with Google’s index. [0]> The fact that we - and companies like Stanford, Nvidia, Adobe, and the United Nations - have had to rely on third-party vendors is a symptom of the closed ecosystem, not a preference. Hopefully someone here can clarify for me, or enumerate some of these “third-party vendors” who seem like they will/might/could be directly affected by this. [0] antibabelic > relevant https://blog.kagi.com/waiting-dawn-search
[1] https://blog.kagi.com/dawn-new-era-search
> [^1]: A note on Google’s existing APIs: Google offers PSE, designed for adding search boxes to websites. It can return web results, but with reduced scope and terms tailored for that narrow use case. More recently, Google offers Grounding with Google Search through Vertex AI, intended for grounding LLM responses. Neither is general-purpose index access. Programmable Search Engine is not designed for building competitive search. Grounding with Google Search is priced at $35 per 1,000 requests - economically unviable for search at scale, and structured as an AI add-on rather than standalone index syndication. These are not the FRAND terms the market needs | | |
| ▲ | tpetry 5 hours ago | parent [-] | | I believe they try to indirectly say they are using SerpApi or a similar product that scrapes Google search results to use them. And other big ones use it too so it must be ok... That must be the reason why they limit the searches you can do in the starter plan. Every SerpApi call costs money. | | |
| ▲ | sixhobbits 4 hours ago | parent [-] | | Google is also suing SerpAPI And I can't prove correlation but they refused to index one of my domains and I think it _might_ be because we had some content on there about how to use SerpAPI |
|
| |
| ▲ | marginalia_nu 6 hours ago | parent | prev | next [-] | | They published this the other day: https://blog.kagi.com/waiting-dawn-search Which saw some discussion on HN. | | | |
| ▲ | monooso 5 hours ago | parent | prev | next [-] | | Kagi does not use Google's search index. From their post which made the front page of HN yesterday [1]: > Google does not offer a public search API. The only available path is an ad-syndication bundle with no changes to result presentation - the model Startpage uses. Ad syndication is a non-starter for Kagi’s ad-free subscription model. [1]: https://news.ycombinator.com/item?id=46708678 | | |
| ▲ | jsnell 4 hours ago | parent [-] | | They then go on to say that they pay a 3rd party company to scrape Google results (and serve those scraped results to their users). So their search engine is indeed based on unauthorized and uncompensated use of Google's index. But since they're not using/paying for a supported API but just taking what they want, they indeed are unlikely to be impacted by this API turndown. | | |
| ▲ | DangitBobby 2 hours ago | parent [-] | | Congrats on saying that in the most one-sided way possible. Google makes it literally impossible for them to pay for access to search results to make the product they want (customizable subscription search with no ads), and Google also is the de-facto globally sanctioned crawler because they are the only search engine anyone gives a shit about, and also sites need to be indexed by them to survive. In short, Google owns the river and sells the boats, and the public built a wall around it. Google is in a monopoly position in search. | | |
| ▲ | nova22033 an hour ago | parent [-] | | >In short, Google owns the river and sells the boats, and the public built a wall around it. That would be a monopoly if there was only 1 river in the whole world. |
|
|
| |
| ▲ | pell 6 hours ago | parent | prev | next [-] | | I think Kagi buys search engine results from SERP vendors who typically scrape Google’s results and offer an API experience on top of it. | |
| ▲ | echelon 6 hours ago | parent | prev | next [-] | | No wonder Kagi is angry. Google is a monopoly across several broad categories. They're also a taxation enterprise. Google Search took over as the URL bar for 91% of all web users across all devices. Since this intercepts trademarks and brand names, Google gets to tax all businesses unfairly. Tell your legislators in the US and the EU that Google shouldn't be able to sell ads against registered trademarks (+/- some edit distance). They re-engineered the web to be a taxation system for all businesses across all categories. Searching for Claude -> Ads in first place Searching for ChatGPT -> Ads in first place Searching for iPhone -> Ads in first place This is inexcusable. Only searches for "ChatGPT versus", "iPhone reviews", or "Nintendo game comparison" should allow ads. And one could argue that the "URL Bar" shouldn't auto suggest these either when a trademark is in the URL bar. If Google won't play fair, we have to kill 50% of their search revenue for being egregiously evil. If you own a trademark, Google shouldn't be able to sell ads against you. -- Google's really bad. Ideally we'd get an antitrust breakup. They're worse than Ma Bell. I wouldn't even split Google into multiple companies by division - I'd force them to be multiple copies of the same exact entity that then have to compete with each other: Bell Systems -> {BellSouth, Bell Atlantic, Southwestern Bell, ...} Google -> {GoogleA, GoogleB, GoogleC, ...} They'd each have cloud, search, browser, and YouTube. But new brand names for new parent companies. That would create all-out war and lead to incredible consumer wins. | | |
| ▲ | thewebguyd 10 minutes ago | parent | next [-] | | > If you own a trademark, Google shouldn't be able to sell ads against you. This is frustrating even from a consumer perspective. Before I ran adblock everywhere, I couldn't stand that typing in a specific company I was looking for would just serve ads from any number of related brands that I wasn't looking for that were competitors. | |
| ▲ | marginalia_nu 6 hours ago | parent | prev | next [-] | | Could probably argued that search access is an essential facility[1], though it doesn't appear antitrust law has anywhere near the same sort of enforcement it did in the past. [1] https://en.wikipedia.org/wiki/Essential_facilities_doctrine | |
| ▲ | throwaway290 4 hours ago | parent | prev | next [-] | | what stops Kagi from indexing internet and makes them pay some guys to scrape search results from Google? one guy at Marginalia can do it and entire dev team at a PAID search engine can't? | | |
| ▲ | mrweasel an hour ago | parent | next [-] | | I don't know about others, but we have special rules for Google, Bing, and a few others, rate-limiting them less than some random bot. The problem is scrapers (mostly AI scrapers from what we can tell). They will pound a site into the ground and not care and they are becoming increasingly good at hiding their tracks. The only reasonable way to deal with them is to rate-limit every IP by default and then lifting some of those restrictions on known, well behaving bots. Now we will lift those restrictions if asked, and frequently look at statistics to lift the restrictions from search engines we might have missed, but it's an up hill battle if you're new and unknown. | |
| ▲ | DangitBobby 2 hours ago | parent | prev [-] | | As we've seen here on HN on the AI boom, it's not wonderful when a bunch of companies all use bots to scrape the entire web. Many sites only allow Google scrapers in robots.txt and the public will fight you hard if you scrape them without permission. It's just one of those things where it would be better for everyone if search engines could pay for access to the work that's done only once. | | |
| ▲ | echelon 2 hours ago | parent [-] | | > Many sites only allow Google scrapers in robots.txt and the public will fight you hard if you scrape them without permission. This just lets a monopoly replace the website instead of distributing power and fostering open source. The same monopoly that was already bleeding off the web's utility and taxing it. |
|
| |
| ▲ | onetokeoverthe 5 hours ago | parent | prev [-] | | [dead] |
| |
| ▲ | 6 hours ago | parent | prev [-] | | [deleted] |
|
|
|
| ▲ | raincole 5 hours ago | parent | prev | next [-] |
| > “search the entire web” TIL they allowed that before. It sounds a bit crazy. Like Google is inviting people to repackage google search itself and sell it / serve with their own ads. |
| |
| ▲ | MrGilbert 4 hours ago | parent | next [-] | | You know, back in the days, the web used to be more open. Also - just because you CAN do something, doesn't mean you HAVE to. | |
| ▲ | whs 4 hours ago | parent | prev | next [-] | | I tried it and contributed to searx. It didn't give the same result as Google, and it also have 10k request rate limit (per month I believe). More than that you'll have to "contact us" | |
| ▲ | shevy-java 4 hours ago | parent | prev [-] | | It basically means that Google is now transitioning into a private web. Others have to replace Google. We need access to public information. States
can not allow corporations to hold us here hostage. |
|
|
| ▲ | saltysalt 6 hours ago | parent | prev | next [-] |
| I built my own web search index on bare metal, index now up to 34m docs: https://greppr.org/ People rely too much on other people's infra and services, which can be decommissioned anytime. The Google Graveyard is real. |
| |
| ▲ | orf 6 hours ago | parent | next [-] | | Number of docs isn’t the limiting factor. I just searched for “stackoverflow” and the first result was this: https://www.perl.com/tags/stackoverflow/ The actual Stackoverflow site was ranked way down, below some weird twitter accounts. | | |
| ▲ | saltysalt 6 hours ago | parent [-] | | I don't weight home pages in any way yet to bump them up, it's just raw search on keyword relevance. | | |
| ▲ | dredmorbius 5 hours ago | parent | next [-] | | Google's entire (initial) claim-to-fame was "PageRank", referring both to the ranking of pages and co-founder Larry Page, which strongly prioritised a relevance attribute over raw keyword findings (which then-popular alternatives such as Alta Vista, Yahoo, AskJeeves, Lycos, Infoseek, HotBot, etc., relied on, or the rather more notorious paid-rankings schemes in which SERP order was effectively sold). When it was first introduced, Google Web Search was absolutely worlds ahead of any competition. I remember this well having used them previously and adopted Google quite early (1998/99). Even with PageRank result prioritisation is highly subject to gaming. Raw keyword search is far more so (keyword stuffing and other shenanigans), moreso as any given search engine begins to become popular and catch the attention of publishers. Google now applies other additional ordering factors as well. And of course has come to dominate SERP results with paid, advertised, listings, which are all but impossible to discern from "organic" search results. (I've not used Google Web Search as my primary tool for well over a decade, and probably only run a few searches per month. DDG is my primary, though I'll look at a few others including Kagi and Marginalia, though those rarely.) <https://en.wikipedia.org/wiki/PageRank> "The anatomy of a large-scale hypertextual Web search engine" (1998) <http://infolab.stanford.edu/pub/papers/google.pdf> (PDF) Early (1990s) search engines: <https://en.wikipedia.org/wiki/Search_engine#1990s:_Birth_of_...>. | | |
| ▲ | saltysalt 4 hours ago | parent [-] | | PageRank was an innovative idea in the early days of the Internet when trust was high, but yes it's absolutely gamed now and I would be surprised if Google still relies on it. Fair play to them though, it enabled them to build a massive business. | | |
| ▲ | marginalia_nu 4 hours ago | parent | next [-] | | Anchor text information is arguably a better source for relevance ranking in my experience. I publish exports of the ones Marginalia is aware of[1] if you want to play with integrating them. [1] https://downloads.marginalia.nu/exports/
grab 'atags-25-04-20.parquet' | | |
| ▲ | dredmorbius 3 hours ago | parent | next [-] | | Though I'd think that you'd want to weight unaffiliated sites' anchor text to a given URL much higher than an affiliated site. "Affiliation" is a tricky term itself. Content farms were popular in the aughts (though they seem to have largely subsided), firms such as Claria and Gator. There are chumboxes (Outbrain, Taboola), and of course affiliate links (e.g., to Amazon or other shopping sites). SEO manipulation is its own whole universe. (I'm sure you know far more about this than I do, I'm mostly talking at other readers, and maybe hoping to glean some more wisdom from you ;-) | | |
| ▲ | marginalia_nu 3 hours ago | parent [-] | | Oh yeah, there's definitely room for improvement in that general direction. Indexing anchor texts is much better than page rank, but in isolation, it's not sufficient. I've also seen some benefit fingerpinting the network traffic the websites make using a headless browser, to identify which ad networks they load. Very few spam sites have no ads, since there wouldn't be any economy in that. e.g. https://marginalia-search.com/site/www.salon.com?view=traffi... The full data set of DOM samples + recorded network traffic are in an enormous sqlite file (400GB+), and I haven't yet worked out any way of distributing the data yet. Though it's in the back of my mind as something I'd like to solve. | | |
| ▲ | dredmorbius 2 hours ago | parent [-] | | Oh, that is clever! I'd also suspect that there are networks / links which are more likely signs of low-value content than others. Off the top of my head, crypto, MLM, known scam/fraud sites, and perhaps share links to certain social networks might be negative indicators. | | |
| ▲ | marginalia_nu an hour ago | parent [-] | | You can actually identify clusters of websites based on the cosine similarity of their outbound links. Pretty useful for identifying content farms spanning multiple websites. Have a lil' data explorer for this: https://explore2.marginalia.nu/ Quite a lot of dead links in the dataset, but it's still useful. |
|
|
| |
| ▲ | saltysalt 4 hours ago | parent | prev [-] | | Very interesting, and it is very kind of you to share your data like that. Will review! |
| |
| ▲ | snowwrestler 3 hours ago | parent | prev [-] | | Google’s biggest search signal now is aggregate behavioral data reported from Chrome. That pervasive behavioral surveillance is the main reason Apple has never allowed a native Chrome app on iOS. It’s also why it is so hard to compete with Google. You guys are talking about techniques for analyzing the corpus of the search index. Google does that and has a direct view into how millions of people interact with it. | | |
| ▲ | saltysalt 2 hours ago | parent | next [-] | | Yes indeed, they have an impossibly deep moat and deeper pockets. I'm certainly not trying to compete with them with my little side project, it's just for fun! | |
| ▲ | xnx 2 hours ago | parent | prev [-] | | > That pervasive behavioral surveillance is the main reason Apple has never allowed a native Chrome app on iOS The Chrome iOS app still knows every url visited, duration, scroll depth, etc. |
|
|
| |
| ▲ | orf 6 hours ago | parent | prev | next [-] | | Sure, but the point is results are not relevant at all? It’s cool though, and really fast | | |
| ▲ | saltysalt 6 hours ago | parent | next [-] | | I'll work on that adjustment, it's fair feedback thanks! | | |
| ▲ | direwolf20 5 hours ago | parent [-] | | Unfortunately this is the bulk of search engine work. Recursive scraping is easy in comparison, even with CAPTCHA bypassing. You either limit the index to only highly relevant sites (as Marginalia does) or you must work very hard to separate the spam from the ham. And spam in one search may be ham in another. | | |
| ▲ | saltysalt 5 hours ago | parent [-] | | I limit it to highly relevant curated seed sites, and don't allow public submissions. I'd rather have a small high-quality index. You are absolutely right, it is the hardest part! |
|
| |
| ▲ | globular-toast 5 hours ago | parent | prev [-] | | What do you mean they're not relevant? The top result you linked contained the word stackoverflow didn't it? It's showing you exactly what you searched for. Why would you need a search engine at all if you already know the name of the thing? Just type stackoverflow.com into your address bar. I feel like Google-style "search" has made people really dumb and unable to help themselves. | | |
| ▲ | orf 5 hours ago | parent [-] | | the query is just to highlight that relevance is a complex topic. few people would consider "perl blog posts from 2016 that have the stack overflow tag" as the most relevant result for that query. |
|
| |
| ▲ | pjc50 4 hours ago | parent | prev [-] | | Confluence search does this, for our intranet. As a result it's barely usable. Indexing is a nice compact CS problem; not completely simple for huge datasets like the entire internet, but well-formed. Ranking is the thing that makes a search engine valuable. Especially when faced with people trying to game it with SEO. |
|
| |
| ▲ | dust-jacket 42 minutes ago | parent | prev | next [-] | | This is mad but cool. Keep at it. | |
| ▲ | jfindley 4 hours ago | parent | prev | next [-] | | Unfortunately the index is the easy part. Transforming user input into a series of tokens which get used to rank possible matches and return the top N, based on likely relevence, is the hard part and I'm afraid this doesn't appear to do an acceptable job with any of the queries I tested. There's a reason Google became so popular as quickly as it did. It's even harder to compete in this space nowadays, as the volume of junk and SEO spam is many orders of magnitude worse as a percentage of the corpus than it was back then. | | |
| ▲ | saltysalt 2 hours ago | parent [-] | | I am definitely not trying to complete with Google, instead I am offering an old-school "just search" engine with no tracking, personalization filtering, or AI. It's driven by my own personal nostalgia for the early Internet, and to find interesting hidden corners of the Internet that are becoming increasingly hard to find on Google after you wade through all of the sponsored results and spam in the first few pages... | | |
| ▲ | prophesi 43 minutes ago | parent [-] | | There may be a free CS course out there that teaches how to implement a simplified version of Google's PageRank. It's essentially just the recursive idea that a page is important if important pages link to it. The original paper for it is a good read, too. Curiously, it took me forever to find the unaltered version of the paper that includes Appendix A: Advertising
and Mixed Motives, explaining how any search engine with an ad-based business model will inherently be biased against the needs of their users[0] [0] https://www.site.uottawa.ca/~stan/csi5389/readings/google.pd... |
|
| |
| ▲ | 1718627440 4 hours ago | parent | prev | next [-] | | The input on the results page doesn't work, you always need to return to the start page on which the browser history is disabled. That's just confusing behaviour. | | |
| ▲ | saltysalt 2 hours ago | parent [-] | | I guess you used the return key instead of clicking on the search icon? Seems to be a bug with the return key, I'll fix that this weekend sorry. | | |
| ▲ | 1718627440 2 hours ago | parent [-] | | True, didn't occur to me, that I should click on the icon instead. Once I have clicked on the search icon once, enter also works. When I input a short query (single letter) it sometimes just shows a blank page, but maybe that is just HNs hug of death. Consider putting the query term more prominently in the front of the URL, so users can edit it. Also from the startpage, the URL in the URLbar isn't updated. As I already wrote, the browser shows completion for the searchbar on the result page, but does not for the one one the startpage. For my taste I would prefer less JS trickery, which would maybe already get rid of some of these issues. | | |
| ▲ | saltysalt a few seconds ago | parent [-] | | Appreciate the detailed feedback! A lot of the JS trickery and URL shenanigans I'm doing is to prevent bot spam attempts, which was a real problem in the beginning. |
|
|
| |
| ▲ | tosti 4 hours ago | parent | prev | next [-] | | This is pretty cool. Don't let the naysayers stop you. Taking a stab at beating Google at their core product is bravery in my book. The best of luck to you! | | | |
| ▲ | 1718627440 4 hours ago | parent | prev | next [-] | | You should consider filtering by input language. Showing the same Wikipedia article in different languages is not helpful when I am searching in English. Also you may unify by entries by URL, it shows the same URL, just with different publish dates, which is interesting and might be useful, but should maybe be behind a toggle, as it is confusing at first. | | |
| ▲ | saltysalt 2 hours ago | parent [-] | | Great feedback, agree I need to filter here. Some website localization is very hard to work around, because they will try to geo-locate the IP address of your bot and redirect it accordingly to a given language... | | |
| ▲ | 1718627440 an hour ago | parent [-] | | The issue I was having was with the query "term+wikipedia" it then shows the wikipedia article in Czech, Hungarian, Russian, some kind of Arab and other before finally showing the English version. Then also a lot of that occur 2,3,4+ times with the same URL, just differing in crawltime by a few minutes. |
|
| |
| ▲ | renegat0x0 6 hours ago | parent | prev | next [-] | | I made also something for my own search needs. It's just an SQLite table of domains, and places. I have your search engine there also ;-) https://github.com/rumca-js/Internet-Places-Database Demo for most important ones https://rumca-js.github.io/search | | | |
| ▲ | johnofthesea 6 hours ago | parent | prev | next [-] | | I tested it using a local keyword, as I normally do, and it took me to a Wikipedia page I didn’t know existed. So thanks for that. | | | |
| ▲ | lolive 5 hours ago | parent | prev [-] | | Lol, a GooglePlus URL was mentionned on a webpage i browsed this week.#blastFromThePast | | |
|
|
| ▲ | salawat 6 hours ago | parent | prev | next [-] |
| It's been clear for the last decade that we have to wean ourselves off of centralized search indexes if only to innoculate the Net against censorship/politically motivated black holing. I can only weep at this point, as the heroes that were the Silent and Greatest generations (in the U.S.), who fought hard to pass on as much institutional knowledge as possible through hardcore organization and distribution via public and University library, have had that legacy shit on by these ad obsessed cretins. The entirety of human published understanding; and we make it nigh impossible for all but the most determined to actually avail themselves of it. |
|
| ▲ | 6 hours ago | parent | prev [-] |
| [deleted] |