I am hard put to defend reddit and a lot of the things they do ... but I can attest, from personal experience, that scrapers are a huge problem for anybody trying to keep a complex site working. Withholding the display of the comments will significantly reduce the database traffic needed to satisfy a request; I can see why they would want to do that for anonymous readers.

▲

inigyou an hour ago | parent | next [-]

Well Reddit would have a lot less scraper traffic if it hadn't shut down the feed API 3 years ago in an attempt to keep its data closed and only distributed to Google (which paid money for it). Self-inflicted. No sympathy.

▲

corbet an hour ago | parent [-]

I doubt it. The scraper botnets aren't going to bother with niceties like APIs, they just come in over HTTPS and grab anything they can.

	▲	inigyou an hour ago \| parent [-]
		There was a whole mini-industry around scraping specifically Reddit using the API feed because it was so open. AI companies first trained on Pushshift, which is like Common Crawl for Reddit, which is why Reddit shut down Pushshift (with legal threats IIRC) at the same time.

▲

NitpickLawyer an hour ago | parent | prev | next [-]

I don't understand why sites like reddit need to do any work on (unauthd) GET. They are not instant messaging platforms, they are forums. You can live with a ~1-2-5 minute delay on forums. So POSTs (from logged in users) append to a queue, and a worker does work at regular intervals, creates the HTML and that gets sent to a cache + distributed to a CDN if needed. There's no need to do work on unauthd GETs for a forum.

▲

uhhhhwhaaaa an hour ago | parent | prev | next [-]

I personally don't like it, but hard to blame them.

Outside some Stoic altruists, this is their advantage and I'd do the same.

▲

rob an hour ago | parent | prev [-]

Agreed; we have a Linode VPS for some small WordPress websites and it was grinding to a halt after an AI crawler started iterating through every possible date in "The Events Calendar"’s URL structure, sequentially scraping calendar pages all the way back to like 1970. I'm sure that's partially my fault for not putting it behind Cloudflare or anything though.

▲

account42 an hour ago | parent [-]

The response to that should be to restrict the events calendar and similar dynamic functionality, not to restrict the 99% of users who are accessing what should be 100% staticly cached HTML.

▲

rob an hour ago | parent [-]

Good idea; I'll make sure we have all random ?date=1970-01-01, ?date=1970-01-02, all the way to ?date=2026-07-22 URLs cached, even the ones that are 404s but the crawlers still try anyways. Thank you!

▲

account42 39 minutes ago | parent [-]

It's always entertaining when people proudly display their lack reading comprehension.

	▲	rob 36 minutes ago \| parent [-]
		That's very true.