Remix clone Hacker News

new | show | ask | jobs Github

	▲	reconnecting an hour ago
		And assume you have User-agent: meta-externalagent Disallow: /
	▲	Symbiote 37 minutes ago \| parent \| next [-]
		I have observed the same from Meta's crawler. `User-agent: * Disallow: /` on e.g. our preproduction site, Meta is the only big-tech crawler that accesses it, at least with an honest user agent. (Meta also accesses disallowed paths on the production site.)
	▲	kev009 an hour ago \| parent \| prev [-]
		They don't obey *, they don't get their own entry. I'd rather just poison their data, it's a well known behavior from them. https://www.reddit.com/r/webdev/comments/1sdzd1q/metas_ai_cr...