by checking which follow robots.txt?