Remix.run Logo
simonw 5 days ago

Direct quote from https://platform.openai.com/docs/bots

"OpenAI uses the following robots.txt tags to enable webmasters to manage how their sites and content work with AI."

Then for GPT it says:

"GPTBot is used to make our generative AI foundation models more useful and safe. It is used to crawl content that may be used in training our generative AI foundation models. Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models."

What are you seeing here that I'm missing?

hooloovoo_zoo 5 days ago | parent [-]

My read is that they are describing functionality for site owners to provide input about what the site owner thinks should happen. OpenAI is not promising that is what WILL happen, even in the narrow context of that specific bot.