Thanks - wasn't aware. (why: certificate-transparency, open-research-datasets, webwidecrawl)

I still don't fathom why they just _ignore_ the request not to be scraped with the above headers. It's rude.