| ▲ | inigyou an hour ago | |
Everyone ignores robots.txt because every robots.txt blocks everyone. | ||
| ▲ | nubinetwork 30 minutes ago | parent [-] | |
I allow Google, Bing, Yahoo, Kagi, and Apple on my personal blog... who else do I need to allow? Claude, OpenAI, Alibaba, and perplexity can go to hell. | ||