How else do you expect them to get the information from your site if you block them from accessing it?

The expectation should be on the LLM to admit they don’t know the answer rather than blame devs for not allowing crawling.

"How do you expect the gangsters to protect your business if you don't pay them?"

In many, if not most cases, the producers of this information never asked for LLMs to ingest it.