I fail to understand how is this different to a google web crawler in practice other than ignoring the robots.txt?