Has it been settled whether robots.txt applies to user-driven chat sessions and if things like the crawl delay should be applied to say an end-user, an ip address, a harness provider, etc? My understanding is robots.txt is more for training exclusions, but less so for agent work.

▲

bityard an hour ago | parent | next [-]

robots.txt applies (or should, in my opinion) to anything that automatically follows a link. Basically any software that is not a human-controlled web browser or single-shot curl command. Everything else: robot.

▲

xena 3 hours ago | parent | prev [-]

AI bros think they should be exempt from robots.txt. Administrators of big services beg to differ. No solid consensus has arisen. I bet it's gonna take a lawsuit or two to see how it shakes out.

	▲	Dylan16807 31 minutes ago \| parent \| next [-]
		wget ignores robots.txt outside of recursive mode. I think it's correct to do so, and I think an AI loading a handful of pages in response to a command should be about the same.
	▲	ghaff 2 hours ago \| parent \| prev \| next [-]
		From the start, robots.txt has always been an indicator of a site's preference with no actual legal significance.
	▲	recursive 2 hours ago \| parent \| prev [-]
		If a new directive was introduced that allows for an explicit setting in robots.txt, do you think the bros would follow it anyway? Something like `ALLOW AGENTS` or `DISALLOW AGENTS`