| ▲ | gooob 3 hours ago | |
wait why not robots.txt? | ||
| ▲ | reconnecting 3 hours ago | parent [-] | |
Good question, at least OAI-SearchBot is hitting robots.txt. I assume the real issue is that what overloads the servers like security bots, SEO crawlers, and data companies — are the ones that don't respect robots.txt in full, but they wouldn't respect LLMs.txt either. | ||