I am pretty sure you have to enable cloudflare to manage your robots.txt, it shouldn't be doing that by itself. Maybe they did it by accident, it is just 1 click.

My understanding is that the current default allows AI training bots - but this actually going to change in 2 months time.

https://developers.cloudflare.com/changelog/post/2026-07-01-...

From Sept 15 all new sites added to CF will even block Googlebot by default on any page that serves ads as I understand it.

I think it's CF trying to force Google to separate out their bot traffic into bots for training and bots for the search index.

I think CF sees a big opportunity to get businesses to pay them to allow certain uses of their data but block others.

They're also starting a registry of "Approved" crawlers.