| ▲ | Groxx an hour ago | |||||||
For efficiently-hosted sites with little media it's not too bad. E.g. hosting a static site just doesn't cost much, even if you're hammered occasionally. That's extremely far from all sites though. It's probably safe to say it's a severe minority, particularly when you ignore personal / non-profit-bringing sites. Tons of small and large sites run stuff like poorly-written wordpress or ruby on rails or thousands of microservices doing god knows what. A major increase in request volume on those can easily mean significant increases in hosting charges (e.g. small-% on big, many multiples on small) or significant effort in optimizing (which is expensive too). | ||||||||
| ▲ | reconnecting 11 minutes ago | parent [-] | |||||||
The website I mentioned has over 15k webpages and ~200 GB of media, and yet we monitor bots manually and only block them if they're pulling 5k requests in a row. Malicious URLs, multiply 404 are blocked by default. HEAD request rejected. Even on a very bad day, the server's page load time doesn't go over 1s. However, it seems like I'm indeed looking at the problem through the wrong prism, as what I've seen from the comments suggests that the initial issue is performance, and the bots are what uncover it. | ||||||||
| ||||||||