| ▲ | jacobn 4 hours ago | |
I just complained to them the other day! They were scraping our weather website to no end, very much including the disallowed path prefixes. Did end up just adding them to our WAF blocklist, which is weirdly ironic - hosting on their infra & using their services to block their AI scraper... | ||
| ▲ | BLKNSLVR 4 hours ago | parent | next [-] | |
I hope you leave it on the WAF. If they're only just deciding to respect robots.txt, which has been internet infrastructure forever, then it's probably still incredibly amateur software with 'Amazon-priorities' rather than 'responsible internet traffic' priorities. | ||
| ▲ | adrianvi 2 hours ago | parent | prev [-] | |
step 1: create the problem, step 2: sell the solution, step 3: profit | ||