I have always blamed Chinese AI operators for the hundreds of thousands of residential proxy requests I receive each day, but there's no way to be sure.

They absolutely dwarf real traffic if left unchecked (I blocked all countries outside my target area).

From what I've seen, when operating in an environment with bash tools, Claude and ChatGPT will react to a site blocking their normal, attributed fetch tool by calling curl or writing a python script with a different user agent to try and bypass the blocks, rather than respecting the site owners' wishes.

It'd be one thing if they required the user to consciously say "yes, I want you to bypass the block" (still unethical), but the fact that it's just default behavior really says something about the service design alignment.

This is not true, Claude and ChatGPT have sent traffic to sites I maintain without any user-agent, it might be they have stopped doing this.

Google has done various versions of this as well, Grok too.

It could be these bots were in some custom harness or something, but they literally sent requests in very weird ways, that humans won't be doing.

I have ended up putting my sites behind vpns or anubis and similar. This is beyond painful.

>Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections

The last part is extremely dubious with some recent reveals.

I've spent a lot of time building datamint.xyz, and dealing with user agents, datacenter IPs, and VPNs gets pretty tricky. What works right now is a fork of the latest Firefox with anti-fingerprinting patches, paired with a high-quality residential proxy network with clean IPs. Even the basic setup needs 11+ code-level patches, and there are still a few edge cases, but this setup should cover 99% of cases.