I've seen this too.

I think it's a few bad actors really. Because nobody serious about indexing content will do what these crawlers are doing..

They are consume lots of content that is unoriginal or duplicate or duplicate with minor modifications.

Not sure how to block, but maybe a little bit of law enforcement could dramatically reduce the number of TVs being used a proxies.

▲

iamniels 4 hours ago | parent [-]

I run a website with 10k unique pages. If I leave the gates open, Meta hits it 200.000 times per day. Every day. What are you paying developers $500k for Mark?

	▲	jopsen 3 hours ago \| parent [-]
		200k req/day = 2.3 req/s That's bad for static content. Try adding search, with pagination and 16 filters that can toggled on off. And 1000 tags, give each page 5.