please add a robots.txt, its quite a d### move to people who build responsible crawlers for fun.

It's a fairly trivial inconvenience. You can just add something to the effect of the below code, and you'll not get stuck and realistically not skip over crawling anything of value.

  if (response_time > 8 seconds && response_payload < 2048 bytes) {
    extract_links = false;
  }