| ▲ | fragmede an hour ago | |||||||
How does HN fare with scraper load? Is it just CDN and pay the extra bandwidth bill for anon hit requests? | ||||||||
| ▲ | Bender an hour ago | parent | next [-] | |||||||
Not dang and not the person you are asking but there is no CDN. HN is just two servers running BSD one active and one standby. HN is all text so there is not much bandwidth usage. I did an experiment and linked from HN to my lame blog site and disabled all my anti-scraping measures. Even with all the bots I did not see that much traffic. I suspect some people are specifically being targeted by very poorly configured or very poorly written archiving scripts. Just one example thread discussing this with someone on HN [1]. Each case of being targeted will require looking at generalized characteristics but most are easy to stop in my opinion and experience. | ||||||||
| ||||||||
| ▲ | dredmorbius an hour ago | parent | prev [-] | |||||||
For one datapoint ... I have a custom HN CSS which includes some formatting of different sets of user accounts. Admins, for example, get orange highlighting and a dragon emoji (for one does not meddle in the affairs of ...). Also included are leaders, which is the one part of my CSS build script which is, or at least was until a few minutes ago, dynamic. Presently HN is returning "sorry" to my curl request. Given that I run that build manually a few times a month, it's not a matter of hitting HN with frequent scrapes. But HN has become increasingly scrape-hostile over time. Back in 2023 I did a crawl of all of HN's front-page daily history (365.25 days/year * 17 years, so about 6,200 requests), to answer a question which had come up about what was/wasn't mentioned in submission titles. That scrape included a delay (probably either 1 or 10 seconds, possibly more, I don't recall which and may have run the fetch directly from the command line), and ran (initially) without issues. I don't think it would fly today. I reported on findings at the time and several times since: <https://news.ycombinator.com/item?id=36078578> <https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...> | ||||||||