Unfortunately, geofencing is becoming more and more common specifically because of the extra bot traffic.

A shop selling goods in $COUNTRY has very little to gain from allowing arbitrary users from any other country. Allowing bots has always been a tradeoff between the potential benefits and the impact on infrastructure, which is why it's reasonable to identify yourself as a bot for the purpose of rate limiting purposes.

A journalist anonymously using a crawler is indistinguishable from a bad actor trying to collect data for its own purposes, some of which may actually be against the interests of the company in question. In particular, a blog or news publication offering knowledge in exchange for ad revenue stands to disappear because that same knowledge can be compressed in a LLM and offered to the end user directly, completely depriving the source of any revenue.

▲

AnthonyMouse 2 hours ago | parent [-]

> Unfortunately, geofencing is becoming more and more common specifically because of the extra bot traffic.

How is geofencing supposed to help? The crawler operated by technical staff in $foreign_country just switches to compromised smart TVs in your own country and you ultimately end up blocking primarily real humans who don't all know how to change their apparent geolocation.

> A journalist anonymously using a crawler is indistinguishable from a bad actor trying to collect data for its own purposes, some of which may actually be against the interests of the company in question.

There is a major way to distinguish them. If you pass a law requiring them to do something, journalists from major publications have legal departments who then order them to follow the law (and then can't do their journalism), whereas the bad actor is either not interested in following laws or is not in your jurisdiction to be subject to them to begin with.

▲

AdrianCX 2 hours ago | parent | next [-]

> The crawler operated by technical staff in $foreign_country just switches to compromised smart TVs in your own country

That costs the crawler money, not many do it and is something that can be fixed over time

banning country and AS numbers for cloud companies is pretty effective in my experience

	▲	AnthonyMouse an hour ago \| parent [-]
		> That costs the crawler money They're for-profit operations. They have money, and it doesn't cost that much. Proof of work is significantly more expensive (but then you put delays on even the users in your own region). > banning country and AS numbers for cloud companies is pretty effective in my experience It's the sort of thing that works when the first person initially rolls it out because the crawlers only bother to do something about it once a threshold percentage of people are doing it, at which point they use the workaround. And then you have people insisting that it works even though it only used to work, and meanwhile you're still blocking actual humans.

▲

za_creature 2 hours ago | parent | prev [-]

The compromised smart tvs in your country were manufactured by someone that could be held liable for operating a "stealth" bot network, or for allowing someone to do so through negligence.

As for journalists, nothing is stopping their legal department from petitioning a judge to issue a temporarily exemption from the law for a specific period of time under a specific written purpose.

▲

lstodd 2 hours ago | parent [-]

> As for journalists, nothing is stopping their legal department

and this is the point.

so people without a 'legal department' are shut off.

	▲	Spivak an hour ago \| parent [-]
		Sure but this probably wouldn't happen like that, what would happen is that when journalists make that petition it creates a real special case in the law available to everyone.