| ▲ | janmo 3 hours ago | |
His robots.txt explicitly allows bots including LLM bots to scrape his site | ||
| ▲ | jefposk an hour ago | parent [-] | |
The LLM scraper bots ignore robots.txt | ||
| ▲ | janmo 3 hours ago | |
His robots.txt explicitly allows bots including LLM bots to scrape his site | ||
| ▲ | jefposk an hour ago | parent [-] | |
The LLM scraper bots ignore robots.txt | ||