▲ | flir 5 days ago | |||||||
> The first isn't worth arguing against: it's the idea that LLM vendors ignore your robots.txt file even when they clearly state that they'll obey it: https://platform.openai.com/docs/bots So, uh... where's all the extra traffic coming from? | ||||||||
▲ | simonw 5 days ago | parent [-] | |||||||
All of the badly behaved crawlers. | ||||||||
|