> There's a real conflict here. I want Google and Bing and DuckDuckGo to crawl my site and send me new readers. But I don't want everyone else strip-mining it.

Kinda sounds like we're missing a peer to peer network here.

Instead of downloading the same data over and over again we can just download it once and then share it.

Wouldn't that be better for everyone involved?

It would also function as a distributed WayBack Machine, in case anything ever happens to the Internet Archive. (Which I think is desperately needed, bot apocalypse aside.)

▲

inigyou 4 hours ago | parent | next [-]

Anyway, Google doesn't send traffic to your site any more. Only important sites and obvious scams seem to get indexed.

▲

NicuCalcea 5 hours ago | parent | prev | next [-]

Common Crawl?

https://commoncrawl.org/

▲

amazingamazing 5 hours ago | parent | prev [-]

The problem there is trust

	▲	econ 3 hours ago \| parent [-]
		You can validate by repeating the work and sign your content. The domain name system is now just a rent seeking scheme. It was great as a temp solution but over time it has deleted more content than preserved. I might in theory be billed for having a country name but I don't pay for a city, street name, house number or postal code. Online you should be able to move your widget shop to widget street. Can bill people who want to live on real estate street or used car street and/or set some requirements.