There is no single answer to this, it entirely depends on the exact service and the exact page

Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections

Meanwhile other services could use a variety of tactics such as simply setting a browser user agent and/or using random data center IPs or residential IPs, etc etc and then the blocking would just come down to whatever the individual page has configured which can actually be pretty hit or miss

▲

bulder 2 hours ago | parent | next [-]

From what I've seen, when operating in an environment with bash tools, Claude and ChatGPT will react to a site blocking their normal, attributed fetch tool by calling curl or writing a python script with a different user agent to try and bypass the blocks, rather than respecting the site owners' wishes.

It'd be one thing if they required the user to consciously say "yes, I want you to bypass the block" (still unethical), but the fact that it's just default behavior really says something about the service design alignment.

▲

gvfsa an hour ago | parent [-]

Claude and ChatGPT are user agents. For obvious reasons, they serve the user, not the websites' owners. (Also, if you don't want people to be able to access some information, then don't put it online.)

▲

matkoniecz 36 minutes ago | parent [-]

> if you don't want people to be able to access some information

Claude and ChatGPT are not people

	▲	pseudosaid 5 minutes ago \| parent \| next [-]
		they are tools. who do you think is using them? dogs? your comment is another classic hacker news comment. missing the meaning to validate themselves. Lower than my own effort, but wow does it come in so many forms on this site. as if mine is any better
	▲	gvfsa 24 minutes ago \| parent \| prev [-]
		Chrome and Firefox aren’t people either.

▲

minraws 2 hours ago | parent | prev | next [-]

This is not true, Claude and ChatGPT have sent traffic to sites I maintain without any user-agent, it might be they have stopped doing this.

Google has done various versions of this as well, Grok too.

It could be these bots were in some custom harness or something, but they literally sent requests in very weird ways, that humans won't be doing.

I have ended up putting my sites behind vpns or anubis and similar. This is beyond painful.

▲

ReptileMan 2 hours ago | parent | prev | next [-]

>Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections

The last part is extremely dubious with some recent reveals.

▲

jack_pp 2 hours ago | parent [-]

recent reveals show that OAI and Anthropic give users the ability to go around bot protections or they internally use it?

	▲	ReptileMan 5 minutes ago \| parent \| next [-]
		I have noticed that agents become extremely creative to fulfill any non refused requests. And then instructions become more of a suggestions.
	▲	amluto 2 hours ago \| parent \| prev [-]
		It’s not exactly recent or a reveal, but I’ve caught at least several OAI models writing Python scripts to access web pages.

▲

m00dy 3 hours ago | parent | prev [-]

I've spent a lot of time building datamint.xyz, and dealing with user agents, datacenter IPs, and VPNs gets pretty tricky. What works right now is a fork of the latest Firefox with anti-fingerprinting patches, paired with a high-quality residential proxy network with clean IPs. Even the basic setup needs 11+ code-level patches, and there are still a few edge cases, but this setup should cover 99% of cases.

▲

thisismyanon2 3 hours ago | parent | next [-]

Thank you for escalating the arms race that's currently destroying the internet. I can't look at the Apple user forums, my health insurance company's portal, pay my county taxes, browse Digikey and Mouser, view Github or whatever forge, etc. etc. without having to jump through Cloudflare or Anubis garbage designed to block your abusive behavior.

I'm just overflowing with joy that you're comfortable wasting my time and money.

▲

stavros 2 hours ago | parent | next [-]

Even if it's not the GP, if there's profit to be made, somebody will do it. I was reminded of the power of economic motives the other day, when it turned out someone bought an entire trustworthy reseller just so they could compromise the Ledger supply chain by including a tiny eSIM module that OCRed the screen and transmitted the seed.

Incentives are crazy powerful.

▲

matkoniecz 34 minutes ago | parent [-]

> if there's profit to be made, somebody will do it

Not always! With sufficiently powerful barriers it can be blocked at least in part.

Shaming awful people is one of available tools.

Or at least not praising them for making small profit at huge costs to others.

	▲	stavros 7 minutes ago \| parent [-]
		Can't argue with doing what we can.

▲

an hour ago | parent | prev [-]

[deleted]

▲

stev1983 2 hours ago | parent | prev | next [-]

Here’s what is says on their site: “Courtesy backoff Adaptive backoff keyed to target response codes. When a target returns 429 or 503, we back off immediately, so our fleet never applies denial-of-service pressure to a site.” By the time you get the error you have already crashed the site. Geez.

▲

latexr 3 hours ago | parent | prev [-]

I have to agree with the (sarcastic) sibling comment. You’re essentially making everyone’s life worse in the name of personal profit.

Please stop. The last thing we need in the world is more greed and selfishness.

▲

lejalv 2 hours ago | parent [-]

Since when have moral appeals worked against the central feature of capitalism?

Only strong regulation can help.

	▲	matkoniecz 31 minutes ago \| parent [-]
		Shaming awful people is one of available tools. Or at least not praising them for making small profit at huge costs to others. Also, it is useful as coordination method, enabling passing laws outlawing some behaviors. (and I am not able to legislate-via-http while posting such comment is easy)