Remix clone Hacker News

new | show | ask | jobs Github

	▲	jacquesm 10 hours ago
		If only we had a file in the / of web servers that you could use to tell scrapers and bots to fuck off. We'd say for instance: `User-Agent: * Disallow: /` And that would be that. Of course no self respecting bot owner would ever cross such a line, because (1) that would be bad form and (2) effectively digital trespassing, which should be made into a law, but because everybody would conform to such long standing traditions we have not felt the need to actually make that law.