Remix.run Logo
thomashabets2 2 hours ago

Something suspiciously absent from this article is addressing whether some of these are in fact human visitors, but humans issuing chatgpt or similar queries instead of going directly.

Is it really a bot if it's in response to a human asking for some aggregate information about charities, triggering a web search and then following the result links to get details for the human? Well, clearly yes it is, but it's a very different proposition from this article's implication that "they have no throttling on their scrapers"[1].

> Challenge 46 datacenter ASNs. Humans don't browse from AWS.

People who have workstations in the cloud do.

> The bots use 99% of the bill and I pay 100% of it.

Running a site this way is always a wallet-DDoS risk.

[1] though yes, by far most will be pure automation with no human in the loop. It's an assumption on my part, but feels like a safe one.

sethops1 2 hours ago | parent | next [-]

> Running a site this way

What do you mean "this way". What other way is there to run the site?

fooey 2 hours ago | parent | prev | next [-]

cloudflares lack of spending caps and and usage alerts is a seriously unnerving approach

bakugo 2 hours ago | parent | prev [-]

> Something suspiciously absent from this article is addressing whether some of these are in fact human visitors, but humans issuing chatgpt or similar queries instead of going directly.

That's not absent from the article, it's right there in the section titled "The Claude ratio". ChatGPT, Claude, etc. use different user-agents for scraping vs user-initiated requests, and the author notes that user-initiated requests were an absolutely miniscule fraction of the total traffic.

thomashabets2 2 hours ago | parent [-]

Oh, that's what "Anthropic's search crawler, had requested 420,680 pages in one week. That same week, Claude sent me 12 human visitors" meant?

Yeah, googling it does seem like "Claude-User" is for user-initiated requests.

By "Claude sent me" I thought the author meant referer header in real browser requests showed that they came from.

I mean, that's what the section "Pages crawled per visitor referred" refers to, right?