I will add a PR to enforce robots.txt before the actual scraping.