Consistency with `robots.txt` seems like a reasonable choice.