Remix.run Logo
input_sh an hour ago

No, Google knows very well about each and every older source, it just refuses to show them.

Here, I can easily prove it: search for something like "site:theguardian.com" and set the date filter to show you results before say 1/1/2020. I guarantee you you won't find a single news article within search results. You will find other pages from their site such as tag pages or old job postings of theirs and what not, but no news article what so ever. Then swap The Guardian for whichever news source you want and do the same.

Then do the same on say DuckDuckGo and it'll work as intended.

weitendorf an hour ago | parent [-]

Are you sure this is Google's doing?

Sites can tell Google what they want indexed and want not to show, and Google's crawler also is supposed to not hammer sites with unwanted indexing, and has quotas where if your site has a massive amount of content but nobody is visiting/searching for some of it, they'll limit how much they crawl/show it.

But this is something site operators want to control for their own purposes; I imagine a news media website gets very little real human traffic (and ad money) to 6 year old news articles, but a lot of bots looking for training data or just archiving their content. Considering news websites aren't loudly complaining about this I wouldn't be so sure it's not their own doing.

an hour ago | parent [-]
[deleted]