Cloudflare are setting themselves up as the arbiter who will decide which requests are a) human, b) authorized AI bots, c) illicit/banned bots.

Given the number of people on HN who report massive problems from scrapers and other bots, it sounds like if Cloudflare doesn't do this, someone else will need to. I might have thought bandwidth was cheap enough now for it not to matter, but I guess the bots are costing some sites a lot of money.

Cloudflare seems to be very excited to eventually get a 30% cut on pay-to-crawl.

As for the bots, I thought the same thing, but it is indeed a huge problem. They've brought my websites down pretty frequently recently. I tried Cloudflare but visitors complained, and I think you can't win against the bots anyway, so I've resorted to performance improvements and serving every request.

It isn't bandwidth that causes problems. It is the various types of load that they can add to your servers. Which is why no centralized vendor can decide the proper caching or what bots should be blocked, allowed, rate limited, etc. Those things do not have standard answers - it depends on what your apps do, your audience, their usage patterns, and sometimes the regulatory environment in which you run.

Cloudflare doesn't block bots. It's trivial to use residential proxies and your very obvious bot will only get blocked maybe 5% of the time when using rotating IPs.