| ▲ | stinky613 4 hours ago | |
A year or two ago I personally encountered scraping bots that were scraping every possible resultant page from a given starting point. So if it scraped a search results page it would also scrape every single distinct combination of facets on that search (including nonsensical combinations e.g. products that match the filter "products where weight<2lbs AND weight>2lbs") We ended up having to block entire ASNs and several subnets (lots from Facebook IPs, interestingly) | ||
| ▲ | chao- 3 hours ago | parent [-] | |
I have encountered this same issue with faceted search results and individual inventory listings. | ||