AI companies are just as interested in stopping competing crawlers as anyone else.