thinking running a website never got you 99.9% of requests from bot is just delusional.

anyway, if you really want to block scrappers, give up Anubis. it just about real users and do nothing against bots. simple block ISN that belong to cloud providers. it's that simple.

▲

dhosek 2 days ago | parent | next [-]

I got to watch traffic shift from cloud IPs to residential proxies in real time after I blocked the cloud IPs. It’s endless whack-a-mole.

What really pisses me off was Meta’s bots ignoring robots.txt when I tried to prevent high levels of traffic hitting meaningless URL variants on a Mediawiki site. Everybody else stopped hitting those URLs but not Meta which was flooding me with so much traffic it was impacting site performance. There source IPs and user agents are now banned.

	▲	viraptor 2 days ago \| parent \| next [-]
		Do not explicitly block that traffic. Return 200 with fake content or 404 and they won't try to workaround it.
	▲	MeshArc 20 hours ago \| parent \| prev [-]
		[flagged]

▲

threetonesun 2 days ago | parent | prev [-]

Some of us deployed websites before bots. Heck once upon a time you had to ask nicely for them to even come visit.