Remix.run Logo
everfrustrated 3 hours ago

I wonder how much of this is traffic caused by peoples agents using web tools causing searches and fetches rather than general trawls of the internet.

corbet 3 hours ago | parent | next [-]

Very little of it. When you see a million IPs systematically working their way through your URL space, it's pretty clear that there's a central control node behind it all.

noxvilleza 2 hours ago | parent | prev | next [-]

Most well-known/large agentic web tools I've seen are actually super honest about who they are -- even when they write out scripts they're very keen to identify themselves using user-agents. Most of the time those tools are fine - it's the ones that happen to have a random choice of the 5 most common Chrome/Firefox user-agents making sequential scrapes but cycling through IPs on African and South American residential IPs that are the problem!

dawnerd 2 hours ago | parent | prev [-]

I've seen some logs where a bunch of random ips were hitting a client's search endpoint feeding what looked like user questions to it. Of course none of them returned anything useful but it was causing a lot of strain and even causing the site to go down (gotta love wordpress's stock search).

I'm guessing the training companies are taking real/synthesized user queries and trying to distill what they can from site searches.