| ▲ | deaton 8 hours ago | |
What you gain? Nothing, but they and other AI companies have decided not to respect your robots.txt | ||
| ▲ | loloquwowndueo 8 hours ago | parent [-] | |
There are other ways to block robots from crawling our sites. I have a robots.txt but place no faith in it, it’s just there because it’s cheap and does stop some of the crawlers. | ||