Remix.run Logo
noxvilleza 2 hours ago

Most well-known/large agentic web tools I've seen are actually super honest about who they are -- even when they write out scripts they're very keen to identify themselves using user-agents. Most of the time those tools are fine - it's the ones that happen to have a random choice of the 5 most common Chrome/Firefox user-agents making sequential scrapes but cycling through IPs on African and South American residential IPs that are the problem!

TurdF3rguson an hour ago | parent [-]

Yes I've seen it. ClaudeBot will gleefully announce itself when it hammers my niche website a million times a day.

noxvilleza a minute ago | parent [-]

At least those bots are easy to block though. I run a niche stats website for an esport and I have no idea why there's loads of residential trawlers/botnets with 10k+ IPs trying to get that data - most of what they scrape is directly available from Valve's APIs.