Why can't crawling be crowd sourced? It would solve ip rotation and spread the load

6510 2 days ago | parent | next [-]

	▲	catlikesshrimp 2 days ago \| parent [-]
		Too bad it doesn't support android. It is much more energy efficient than anything else I can spare (for 100% uptime contribution)

Poomba 2 days ago | parent | prev | next [-]

That’s how residential proxies work, in a perverse way

2 days ago | parent | prev | next [-]

[deleted]

Common crawl sort of serves this function. I use it. It's a really good foundation.