| ▲ | kazinator an hour ago | |
robots.txt can used as a reward mechanism. If an automated visitor that we would normally block begins by accessing robots.txt that's a good sign and we give that a plus score or whatever. If subsequent accesses from that visitor comply with robots.txt, we let them continue (perhaps with some rate checks). Conversely, if an obviously automated visitor does not fetch robots.txt, that can be an instant ban, kind of like a reverse honey pot, if you will. So it can have some nonzero value in the overall context; it's just silly to think of it as a blocking mechanism. In short, robots.txt documents what access is allowed; it states the policy. Enforcing it the job of something else. Someone you work for in this capacity shouldn't ask about robots.txt because that's the sort of detail for which they have you to deal with. | ||