| ▲ | reconnecting 10 hours ago | |
I believe there is a slight misunderstanding regarding the role of 'AI crawlers'. Bad crawlers have been there since the very beginning. Some of them looking for known vulnerabilities, some scraping content for third-party services. Most of them have spoofed UAs to pretend to be legitimate bots. This is approximately 30–50% of traffic on any website. | ||