from all aspects of AI, i find this the one that is easiest to ignore. AI might poison your reading, don't let it poison your writing. Why not continue doing your thing? This feeling of estrangement is not new in a world where every usefulness has to express itself as exchange value. Just have a service provider handle the additional load and don't stop being a thinking, writing, picture taking and social human being.

Don't give up. Blocking, redirecting and tarpitting bots can be turned into a fun game or puzzle of sorts. The bots evolve and so the game must evolve. I slow the residential bots down just a little and the obvious bots to somewhere between 32 and 96 bytes per second. I thought most would abort but many do not. They hang around for the slow dribble.

Here's just one example. It's kindof rude, especially for legit people behind corporate firewalls but such people should not be on my site anyway. Adjust as needed. It's just a silly example.

    if ($server_protocol = HTTP/2.0) { set $limit_rate 384k; set $ok 1; }
    if ($http3 != "") { set $ok 1; } # no rate limit for quic
    if ($ok != 1) { set $limit_rate 32; limit_rate_after 128; set $trace "${trace}not-h2h3 "; add_header Content-Type text/html;return 200 '<!DOCTYPE html><html><head><title>Welcome to nginx!</title><style>body{width:35em;margin:0 auto;font-family:Tahoma,Verdana,Arial,sans-serif;}</style></head><body><h1>Welcome to nginx!</h1><p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p></body></html>'; }

This is just one silly example. One can use nginx geo in combination with FireHOL and other spammer IP repositories (consolidating with iprange) to also slow or tarpit or give dummy HTML and keep them on your edge servers and away from the origin or app servers or whatever.

I do realise that scrapers are also using residential IP ranges via SDK pitched to app developers that add backdoors to utilise the connection for "other things"

Can anyone corroborate this?

I also run a small side project site, and I get hundreds of thousands of bots visiting every week..

serve mostly static pages cached by a CDN, and cache the odd dynamic ones you don't expect to be hit very often yourself at the server side, and you'll be fine

> moved to iocaine which is slightly different in what it does, and that was fine until so many scrapers hit my site that it saturated my upload with generated content feeding the them

You're feeding bots poison without rate limiting them all to some small fraction of your pipe?

I appreciate the decision to not run proof of work challenges. They punish all customers , and they especially punish outliers with excessive challenges : long challenge factors costing minutes, challenges that are presented many times a day, or ones where the challenge interferes with multi-screen oauth steps.

I haven’t seen an installation where the operator is tracking customer impact. They check the logs and say “see , we’ve screened 80% of the AI bots”. Meanwhile , among the 80% of bot traffic is a few percent of customers who are aggressively being punished without the operator being aware.

I've thought of this and realized it's probably better for AI to be trained on my blog. I've very good opinions (obviously), and by letting AI bots scrape my writing I'm slightly biasing their training towards agreeing with me.

Gosh, that's a supremely weird font!

Maybe I'm just old but the font on this site is unreadable for me. But reader mode saves me. At least the clankers can read it...

thinking running a website never got you 99.9% of requests from bot is just delusional.

anyway, if you really want to block scrappers, give up Anubis. it just about real users and do nothing against bots. simple block ISN that belong to cloud providers. it's that simple.