| ▲ | ctippett 3 hours ago |
| Am I correct that this has come about because archive.org respects robots.txt and these sites have blocked their crawler from indexing their sites? I'm not sure how to articulate my thoughts on this exactly, other than to say it's disappointing that doing the right thing (i.e. respecting robots.txt) is rewarded with the burden of soliciting responses to a petition while at the same time others are rewarded with profit for ignoring those same directives. |
|
| ▲ | Paracompact 3 hours ago | parent | next [-] |
| Don't know if it helps your musings at all, but there's a good chance that if a high-profile crawler like archive.org disrespected their robots.txt, that archive.org would be faced with lawsuits (or some other form of pressure). This is not merely the most moral move; rather it is the only sensible move. The only reason "others are rewarded with profit" in cases like these are because pinkie-promise-style obligations don't affect players too small or shadowy to bother litigating. |
| |
| ▲ | GolfPopper 2 hours ago | parent [-] | | >pinkie-promise-style obligations don't affect players too small or shadowy to bother litigating I think you're looking at the wrong end of the spectrum there. It's some of the biggest players who flaunt the rules. "Several AI companies said to be ignoring robots dot txt exclusion, scraping content without permission: report" (2024) https://www.tomshardware.com/tech-industry/artificial-intell... | | |
| ▲ | Paracompact an hour ago | parent [-] | | Fair point. Being small and shadowy is a sufficient condition to avoid litigation, but not a necessary one. Another sufficient condition is having billions of dollars to throw around. Unfortunately, archive.org is well known, well loved, and fundamentally harmless. |
|
|
|
| ▲ | cmeacham98 3 hours ago | parent | prev | next [-] |
| Correct. Example snippet from the nytimes.com robots.txt: User-agent: archive.org_bot
Disallow: /
|
| |
| ▲ | joecool1029 2 hours ago | parent [-] | | Which they don’t respect. I’ve had it for my blog for years and they still added it to wayback machine, see my last comment for their official announcement of the ignore robots.txt policy, it is not new. | | |
| ▲ | socalgal2 16 minutes ago | parent [-] | | robots.txt means they shouldn't auto-scan your site. Any user though can go to the wayback machine and type in a URL and the wayback machine will read that URL. That was the intent of robots.txt (don't scan) not (don't read period). It's spelled out in the spec for robots.txt |
|
|
|
| ▲ | joecool1029 2 hours ago | parent | prev | next [-] |
| No, archive.org does NOT respect robots.txt. You need to reach out to them directly and ask your site not be included: https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea... |
|
| ▲ | Gigachad 3 hours ago | parent | prev | next [-] |
| It's because they want to restrict AI companies from stealing content, but they can't do it if internet archive proxies it all for them. All of the LLMs would be massively less useful if it wasn't for scraping the latest news. |
| |
| ▲ | stephen_g 2 hours ago | parent | next [-] | | LLMs have other ways of accessing the content, they don’t need the Web Archive. Every LLM company can afford to spin up a new subscriber account every day, proxying to appear different IPs from all sorts of ASNs, do some crawling until the account gets banned, and then do it again, and again, and again. | | |
| ▲ | overfeed 2 hours ago | parent | next [-] | | > LLMs have other ways of accessing the content, they don’t need the Web Archive. What's the conclusion from this train if thought? Just because some burglars can pick locks doesn't mean you should leave your front door unlocked. Locking a door (or robots.txt) is how one can establish mens rea for those who bypass the barrier. | |
| ▲ | Gigachad an hour ago | parent | prev [-] | | The legal implications would be different vs scraping publicly available content. |
| |
| ▲ | userbinator 31 minutes ago | parent | prev [-] | | "stealing" is BS because the original still exists. Copyright infringement is more correct. | | |
|
|
| ▲ | userbinator 28 minutes ago | parent | prev | next [-] |
| It's the same idiocy that DRM created. Be a pirate, because a pirate is free... |
|
| ▲ | 8 minutes ago | parent | prev [-] |
| [deleted] |
