It’s less weird if you think there’s a difference between good bots and bad bots. They can provide services for good bots to use while helping people keep out the bad ones.

If a bot is simulating mouse movement but doing it badly then that’s a strong signal of shenanigans. A good bot will obey robots.txt and do nothing to hide that it’s a bot.

▲

pryelluw 4 days ago | parent | next [-]

Who gets to decide what is a good bot?

▲

pythonaut_16 4 days ago | parent | next [-]

Presumably the site owner with Cloudflare providing enforcement.

Generally isn't a good bot one that respects robots.txt and is respectful of the site's resources by not being spammy?

▲

wnevets 4 days ago | parent | prev | next [-]

by checking which follow robots.txt?

▲

wmf 4 days ago | parent | prev | next [-]

According to their plan, the good bots pay for scraping.

▲

overfeed 4 days ago | parent | prev | next [-]

> Who gets to decide what is a good bot?

Paying Cloudflare's tolls => good bot.

▲

nullpoint420 4 days ago | parent | prev | next [-]

Cloudflare, apparently.

▲

mark212 4 days ago | parent [-]

no, their customers. Why do you assume that people who run websites are clueless and lack agency in this?

	▲	nullpoint420 4 days ago \| parent [-]
		inb4 "we're updating our Terms of Service - all Cloudflare-based scrapers will now automatically be enrolled to scrape your cached websites" and they'll explain how it's great because it's only reading the cached versions, not hitting upstream.

▲

tccole 4 days ago | parent | prev [-]

Me… obviously

▲

mcmcmc 4 days ago | parent | prev [-]

The problem is they have a strong incentive to consider any bot they get paid to host as a good bot, and they can bully competitors by declaring bots they host as bad bots

▲

skybrian 4 days ago | parent [-]

Arguments based on incentives can prove anything. We could just as easily say they have incentives to provide tools to both sides and let them fight it out.

It would be like Gmail automatically whitelisting email from other Gmail accounts or blacklisting email from competitors. Why should Google do that? Their customers are mostly strangers to each other and they want spam filters that work well.

▲

mcmcmc 4 days ago | parent [-]

> It would be like Gmail automatically whitelisting email from other Gmail accounts or blacklisting email from competitors. Why should Google do that? Their customers are mostly strangers to each other and they want spam filters that work well.

Crazy to post this when Google basically does do that with Gmail addresses. They may shove it into your “other” folder but they do absolutely nothing to stop free Gmail accounts from spamming and phishing. Gmail spam is the biggest threat by volume I’ve seen in years of managing email security for clients

	▲	skybrian 4 days ago \| parent [-]
		Huh, I've never seen that.