Remix.run Logo
flir 5 days ago

> The first isn't worth arguing against: it's the idea that LLM vendors ignore your robots.txt file even when they clearly state that they'll obey it: https://platform.openai.com/docs/bots

So, uh... where's all the extra traffic coming from?

simonw 5 days ago | parent [-]

All of the badly behaved crawlers.

flir 5 days ago | parent [-]

Yeah, I read the rest of the conversation and tried to delete. I understand your point now. Apologies.