> This does nothing against your ability to scrape the web the Google way

I thought that Google has access to significant portions of the internet that non-Google bots won’t have access to?

Their crawler has known IPs that get a white-glove treatment by every site with a paywall for example