> Honors robots.txt directives, including crawl-delay

Sounds pretty useless for any serious AI company