> … don't block crawlers.

this rhymes a lot with gangsterism.

if you don’t pay our protection fee it would be a shame if your building caught on fire.

How else do you expect them to get the information from your site if you block them from accessing it?

	▲	robbomacrae 2 days ago \| parent \| next [-]
		The expectation should be on the LLM to admit they don’t know the answer rather than blame devs for not allowing crawling.
	▲	kedean 2 days ago \| parent \| prev \| next [-]
		"How do you expect the gangsters to protect your business if you don't pay them?" In many, if not most cases, the producers of this information never asked for LLMs to ingest it.
	▲	a day ago \| parent \| prev [-]
		[deleted]