| ▲ | andrew_zhong 2 hours ago | |
We do respect robots.txt production - also scraping browser providers like BrightData enforces that. I will add a PR to enforce robots.txt before the actual scraping. | ||
| ▲ | plastic041 2 hours ago | parent [-] | |
How can people believe that you are respecting bot detection in production when your software's README says it can "Avoid detection with built-in anti-bot patches"? | ||