When it comes to residential IPs, that you mentioned, these can only be afforded by scrapers that were specifically made for your website and have a financial incentive. I don't believe that someone would spend money on residential IPs just to crawl the entire internet.

Browser/IP impersonation bots come from DC network, and there are a dozen or so ASNs where they typically live.

General crawlers, from SEO, search engines, meta, alibaba, etc, usually follow robots.txt

The result: the real pain is only the first category, where data from your website has some financial value. But this isn't an infinite number of bots — depending on the business, they're countable amount.