Remix.run Logo
gooob 3 hours ago

wait why not robots.txt?

reconnecting 3 hours ago | parent [-]

Good question, at least OAI-SearchBot is hitting robots.txt.

I assume the real issue is that what overloads the servers like security bots, SEO crawlers, and data companies — are the ones that don't respect robots.txt in full, but they wouldn't respect LLMs.txt either.