> My issue is that crawlers aren’t respecting robots.txt
Cloudflare has a toggle switch to automatically block LLM's + scrapers etc:
https://blog.cloudflare.com/declaring-your-aindependence-blo...