| ▲ | dspillett 15 hours ago | |
> that bother to run JS as crawlers typically don't I'd be surprised if many don't run basic JS, given how many sites require it for basic content these days (load base site, then make a fetch request for the actual content), unless there are less such sites than I think. > One thing I've seen that is interesting'ish is bots failing as they answer the challenge from a different IP than it was requested from, and the source IP is validated as part of the response so they fail. From what little I've looked into the details, it seems that scrapers (both old and new) using a scattering arrangement (built in or via an outgoing proxy) seem to be split 75/25 between those that completely scatter (following requests will almost certainly come from a different address) and those that stay on the same address for at least the first few requests (which would cover login, bot challenge, or such) then move around to avoid requests-per-source-address based resource management measures. Only really simple ones (someone running a recursive wget without proxy) stick to the same source address all the time now. Unfortunately you can't be more strict about changing source addresses because of how many false positives you will see. CGNAT arrangements often move people between addresses fairly regularly, and even without that there is the common case of people moving away from a WiFi AP and automatically flicking to their mobile network (or another nearby AP). | ||