Mostly economies of scale. They have enough data to do the statistics to identify bad actors, and enough bandwidth and servers to facetank the request volume.

It's a cognitive tax on all the parties involved. It works on buyers (websites that pay for it) much the way that llm's work on their users, by working up to the point at which the buyers can't discern it working.

As a user (bot or flesh) you have to jump through these hoops, and to the website operators they get fewer complaints when users (and bots) are able to contort themselves in predictable ways. Versus you'd perceive as a website operator, if your blocking worked at all, that you'd risk seeing a precipitous dropoff of bots if you did things in bespoke ways, as both bots (true positives, among dd and cf survivors) and people (false positives) get jammed.

The net effect in this model would be that such things as cloudflare and datadome take money from websites in order to block "dumb" bots and enfranchise "smart" bots to the point of creating a market for bot services. People that want to bot have to use AI or go to bot service providers.

For website operators that's a worthwhile exchange for dollars, in that dumb bots can cause a lot of harm in wasted compute. Bots are typically dead end users who won't buy ever anyway, or merely inefficient way of doing business if they scrape excessively. And many do scrape excessively.

"reform school for bots" is a good mental model. For this to work, the "reforms" have to be consistent so that bots can conform to them.

It's a peculiar equilibrium and I am not sure that it's all that stable as an equilibrium. Would be a good subject for a formal treatment, game theory and economics.

Would love if they did but most end up just reselling cloudflare. Example WPEngine.

Cloudflare is probably one of the most popular, reasonably priced for general public. You have fastly, akamai, cloudfront as other proxy providers and CDNs with similar capability but higher price point mostly catering to enterprise customers.

Bunny.net is another one that I find similar to Cloudflare from pricing and functionality perspective

TikTok, Instagram, X, etc. These platforms will host you for free and will distribute updates to people, including potential customers via the algorithm.

[flagged]

Not it the site can't stay up because it's constantly being hammered by bots.

Dude, I had a 1Gb of nginx logs daily then I blocked them and it dropped to 10mb weakly.