| ▲ | mitxela 2 hours ago | |
In this day and age, only good bots, the ones you actually want, obey robots.txt. You obtain no benefit by having a restrictive one. | ||
| ▲ | arjie 2 hours ago | parent [-] | |
That is true, but if you have Anubis you presumably do intend to `Disallow` all bots. | ||