People often use specific user agents in there, which is hard if you don't know what the user agents are in advance!

That seems like a potentially very useful addition to the robots.txt "standard": Crawler categories.

Wanting to disallow LLM training (or optionally only that of closed-weight models), but encouraging search indexing or even LLM retrieval in response to user queries, seems popular enough.

▲

6 hours ago | parent | prev | next [-]

[deleted]

▲

wat10000 11 hours ago | parent | prev [-]

If you're using a specific user agent, then you're saying "I want this specific user agent to follow this rule, and not any others." Don't be surprised when a new bot does what you say! If you don't want any bots reading something, use a wildcard.

	▲	lxgr 8 hours ago \| parent \| next [-]
		Yes, but given the lack of generic "robot types" (e.g. "allow algorithmic search crawlers, allow archival, deny LLM training crawlers"), neither opt-in nor opt-out seems like a particularly great option in an age where new crawlers are appearing rapidly (and often, such as here, are announced only after the fact).
	▲	simonw 10 hours ago \| parent \| prev [-]
		Sure, but I still think it's OK to look at Apple with a raised eyebrow when they say "and our previously secret training data crawler obeys robots.txt so you can always opt out!"