Unless you have an idea to stop scraping, the status quo is all downside. Everything gets grabbed but also crawlers overwhelm servers.