Anubis is fine if you want to block bots that are bothering you, but I never understood why they don't have robots.txt at the root level too. I happily obey that for my private crawler. I suppose you don't want to duplicate implementations. Fair enough. I guess I should add an Anubis detector so I can just blacklist because that person clearly doesn't want bots.

In this day and age, only good bots, the ones you actually want, obey robots.txt. You obtain no benefit by having a restrictive one.

One of the standard library rules enables robots.txt passthrough by default. I can't control people not choosing to use it.