> After 15 minutes of confusion, it turned out Cloudflare had put a crazy robots.txt on my site without my consent (Cloudflare, love you guys, but this needs to stop).

Might be the first time I see someone complain about their website being protected from a scraper, instead of the other way around.

▲

sva_ 4 hours ago | parent | next [-]

I am pretty sure you have to enable cloudflare to manage your robots.txt, it shouldn't be doing that by itself. Maybe they did it by accident, it is just 1 click.

	▲	zx76 3 hours ago \| parent [-]
		My understanding is that the current default allows AI training bots - but this actually going to change in 2 months time. https://developers.cloudflare.com/changelog/post/2026-07-01-... From Sept 15 all new sites added to CF will even block Googlebot by default on any page that serves ads as I understand it. I think it's CF trying to force Google to separate out their bot traffic into bots for training and bots for the search index. I think CF sees a big opportunity to get businesses to pay them to allow certain uses of their data but block others. They're also starting a registry of "Approved" crawlers.

▲

nibbleyou 4 hours ago | parent | prev | next [-]

I think the issue is the lack of consent. Whether a service I use is protecting my website from scrapers or feeding everything to scrapers, some of us would prefer that it takes our informed consent before doing so.

	▲	dannyw 2 hours ago \| parent \| next [-]
		Cloudflare is explicitly a service for dropping requests, whether it’s DDoS attacks, as a WAF, or AI crawlers. It offers a lot more too, but this isn’t Cloudflare overstepping imo. FWIW, I just set up a domain last week, and the web UI asked if I want to block AI crawlers or not. Perhaps OP set it up agentically, and the agent didn’t pass an optional param correctly, or ticked the box for him?
	▲	bfjvibybd6cuvu6 4 hours ago \| parent \| prev [-]
		It does.

▲

el_io 3 hours ago | parent | prev | next [-]

You have to enable this, or atleast was the case when I tried less than 1 month ago.

	▲	dannyw 2 hours ago \| parent [-]
		You still have to enable this as of one week ago. I suspect OP might have agentically set up a new Cloudflare domain, and who knows what the agent did during onboarding.

▲

rgrieselhuber 3 hours ago | parent | prev [-]

It should still be done with consent. Robots.txt files don’t protect against determined scrapers anyway.

	▲	adrian17 3 hours ago \| parent [-]
		I thought bot protection was one of reasons for using Cloudflare in the first place (next to general CDN hosting)? After all, they do show a captcha-like challenge on some websites, so I thought that even without robots.txt, it still would have prevented the automated request by default.