Yeah, robots.txt is a great herald example of the type of solution invented by people who don't understand incentives whatsoever.