Fail2ban doesn't scale well to these volumes of traffic and request patterns.

Just like fail2ban is not very useful against a DDOS attack where each unique IP only makes a few requests with a large (hour+) delay in between requests. There is no clear "fail" in these requests, and the fail2ban database becomes huge and far too slow.

- 400,000 Unique IP addresses

- 1 to 3 requests per hour per IP addresses - with delays of over 60 minutes between each request.

- Legit request URLs, legit UA & referrer

Maybe Anubis would help, but it's also a risk for various reasons.