I get the impression that they're just too lazy or incompetent, or in too big a hurry, to put some sensible logic in their scrapers. Maybe they have an LLM write the scraper and don't bother to ask for anything more than "Make a web scraper that gets all the files it can as fast as it can."

The last one I blocked was hitting my site 24 times/second, and a lot of them were the same CSS file over and over.