▲ | bangaladore 11 hours ago | ||||||||||||||||
As this is a robots.txt, it's still not enforced. So how much good can this really do? | |||||||||||||||||
▲ | jsheard 11 hours ago | parent [-] | ||||||||||||||||
If nothing else, it might provide a more consistent way to signal to the crawlers which do respect robots.txt. For example Google, Bing and Apple already offer ways to signal that your site approves of search indexing but not training, but they each require a different non-standard signal for the same thing. For the crawlers that ignore robots.txt nothing changes of course, and for the ones which claim to support a training opt-out you just have to take them at their word that it actually does anything. | |||||||||||||||||
|