Aah shit, completely overlooked the robots.txt functionality. Added it now and made the AI a bit more creative.

Thanks for the feedback!