It is. They will most likely eventually need to move to a walled model for Wayback due to scraper aggressiveness (like Reddit deprecating anonymous old.reddit.com), or behind Cloudflare for aggressive bot and scraping protection. Hard to defend against abuse of a public resource when its intent is public access with as little restriction as possible.

https://en.wikipedia.org/wiki/Tragedy_of_the_commons

(no affiliation)

▲

ronsor 7 hours ago | parent [-]

Reddit has no excuses for the anonymous old.reddit.com removal; they're simply greedy.

On the other hand, the Internet Archive is a non-profit offering a free public resource.

▲

toomuchtodo 7 hours ago | parent [-]

Examples provided as technical examples, strong feelings are out of scope for this thread.

▲

itintheory 7 hours ago | parent [-]

As someone who operates a large non-profit public data driven website, I have some VERY strong feelings about scrapers. We looked into various commercial solutions (Datadome, HUMAN) and based on our traffic estimates from logs we'd be looking at at least 250k/yr for bot mitigation. Anubis is offering a temporary reprieve, but after reading the recent kernel.org article [0] it's increasingly clear that this is a temporary bandaid.

The cheapest solution is to require a login and rate limit by API key. I also have strong feelings about the tragedy of the commons.

[0] https://people.kernel.org/monsieuricon/creepy-crawlies

	▲	toomuchtodo 6 hours ago \| parent [-]
		No strong feelings here is what I meant. Certainly, that energy is best directed into aggressive countermeasures and defense in depth of public goods. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...