While we're dealing with the same issue at work, I sometimes still wonder exactly who these scrapers are. OpenAI, Google, Anthropic and others are normally fairly well behaved (Minus Anthropic attempting to hide behind a browser-for-hire company). Mostly you can get IP range and user-agents for the large players, while it problems mostly stem from bots pretending to be Chrome.

Our largest offenders seems to be mostly limited to South-East Asia, so probably mostly Chinese AI projects, but that's speculation. I also don't recall ever seeing Grok IP ranges or a specific Grok UA, but that doesn't mean that they're hiding, perhaps they're just not interested.

▲

marginalia_nu 2 hours ago | parent | next [-]

Most of the nonsense comes through botnets/residential proxies, so it's very hard to say.

▲

shevy-java 2 hours ago | parent | prev [-]

Well, some websites claimed China is behind it, which could make sense (I would not know either way). At the same time, though, I kind of doubt your carte blanche here for all those companies. Why would you think none of them are responsible for the AI slop spam?

> Our largest offenders seems to be mostly limited to South-East Asia, so probably mostly Chinese AI projects, but that's speculation

Ok. So you also don't know. Well, I don't know either, but I don't make a speculation by claiming x, y, and z companies to be exempt. In my book they are all responsible.

	▲	coldbrewed an hour ago \| parent [-]
		We see constant abuse from the Tencent ASN/associated ACE ASN, and I've memorized the china169 backbone asn as AS4837 because of thr nonstop crawlers splattered across their network ranges. It's not possible to ID the operator running the crawlers running from these networks but there's a clear signal of the origin of some of these entities.