| ▲ | weitendorf 3 hours ago | |
Are you sure this is Google's doing? Sites can tell Google what they want indexed and want not to show, and Google's crawler also is supposed to not hammer sites with unwanted indexing, and has quotas where if your site has a massive amount of content but nobody is visiting/searching for some of it, they'll limit how much they crawl/show it. But this is something site operators want to control for their own purposes; I imagine a news media website gets very little real human traffic (and ad money) to 6 year old news articles, but a lot of bots looking for training data or just archiving their content. Considering news websites aren't loudly complaining about this I wouldn't be so sure it's not their own doing. | ||
| ▲ | 3 hours ago | parent [-] | |
| [deleted] | ||